When Spinning Plates Crash

JEE was not made for the working man. 

I mean, if you look at intro docs[0] you are going to see things like custom deployment descriptor capability. In practice what this means is that they expect a deployment person to take an existing working application, in a custom zip file, open the same file that is in charge of the whole container (how it is deployed, how things are named, what classes are and are not loaded) edit incredibly ill-commented xml files, and expect it to work in a new environment straight away. 

No. JEE was made by corporations with agendas. BEA, Oracle, IBM etc.. each with their own "value added" piece. Specifications written by corporations are nothing more than joint advertising efforts for their products.

An anonymous grey beard from the early cocaine-80's days of selling computer hardware told me once he went to a demo with a machine the size of a room that only had their tech lead inside with a small system to fake out the real thing. For $3 million dollars they wanted this thing. A guy in a box literally hard-wiring answers. 

I spent all week on getting each production version it's own building release branch. I'm still not done. 

• It's maven's SNAPSHOT. 
• It's svn. 
• It's ongoing poor deployment practices. 
• It's slow corp machines.
• It's corrupted outlook pst files.
• It's rushed integrations with no real transaction support.
• It's huge static custom css
• It's 100 degree heat walking to and from work
• It's interviews taking 90 minutes.
• It's bamboo's not latest branch version but current repository version.
• It's the COMPLETE LACK OF AUTOMATED TAPE BACKUPS.
• the list goes on and on and on and on

[0]: We are on jee5 still... so.. 

When You Bungle a Deployment

Ounce of Prevention

As senior reviewer of code, it is my responsibility to review all incoming changes for possible performance hits.

Obviously bad:
  session.addAttribute(new UUID(), new LargeXMLTree())
not so obviously bad (but essentially equivalent)

    < ... rendered="#{conditionforsomelargepage}">
      <a4j:include viewId="#{somelargepage}" />
    </ ... >

For new code, I made sure that the somelargepage was blank.xhtml if rendered was false. I didn't change the old pages. What I didn't realize (this release took 8 months from inception to get out the door, I forgot some changes alright?) was that old pages used to be included like

    < ... rendered="#{conditionforsomelargepage}">
      <ui:include src="#{somelargepage}">
    </ ..>

Which isn't bad at all. Unless you have ui:debug defined anywhere, in which case it is.

Pound of Cure

It's hard to tell your servers are supersaturated. The code worked fine before. We have not enabled new UI features yet. It never came up in test. The logs show different errors each time a server goes down. Servers are going down once every few hours. What happened?

We have no response time analytic capabilities. We have no baseline for time spent in gc. We have no way of telling session sizes in prod. We have no way of telling if this is a rogue action or a rising tide. We have no way of telling if this is a app memory bug, or a server configuration. We have no load testing capabilities.

Nightmare. We had hints though. The servers would be fine with no errors for a few hours until west coast people signed on. The logs would show nothing except for db timeouts of very basic queries. It happened around the same time each day. Eventually we rolled it back. Time wasted: 1 week.

Wasted Effort

But we got out of it. Idea one was to write a service to determine session size. *Heavily* modified jamm was the answer here.

This was a waste of time and did not contribute to final solution. Hey, I have a jsp that I can thread into a war now that will give you session size though. Not all bad. Time wasted: 1 week.

Repro Found

Step two was to write selenium tests from scratch. 1500 lines of code and one hijacked OCR server later, we were able to reproduce the same server behavior. Time spent: 1 week.

Using a trial version of YourKit, I spent 6 hours running the same selenium test case and taking snapshots. By the numbers: new code spent 156% the memory per session that the old code did. New code did not change app memory used (retained - session). I cannot stress how much this overall view was needed to prove that it wasn't a leak, or application memory bug.

Solution

After you can reproduce a problem, it becomes almost trivial to fix. You can literally just tweak at random until you see the problem go away.

Of course, I knew of an optimization beforehand, having it applied it to the new code. So the fix was not random.

Post Notes

I wrote many emails about how our testing infrastructure had holes that let this problem go though. I wrote many more emails about the money and time costs of fixing the infrastructure, and the pros and cons of different ways of fixing our software development life cycle.

Overall, it was one of the more interesting tasks I've had, as "proving" is in my DNA as a maths major.

Final lesson: large JSF apps simply do not scale.

The nothingness abyss referred to as Richfaces 3

This blog really didn't turn out to be about math and programming did it????

Well, let's change that. 

I'm going to talk now about richfaces 3.3.x. [0]

This is what you need if you are programming in richfaces 3: [0]

That's it. If that's down, you can't program in richfaces 3. 

Other quirks I've learned while using it for the past 3 years:

1. a4j has bad xml tree code. like, who hired that *** excuse for a *** *** * and ***?
2. If you are going to open a new browser window, and expect both windows to be used, put the code for the new window in a different war. 
3. richfaces programming style is jquery 1.3, but the thought process is pre-1.3. In case you forgot, or were still in diapers, jquery 1.3 was when they invented feature detection [1]. Thus trying to support any browsers that did not exist in 2009, you are going to have a Bad Time.
4. As far as I can tell JSF2 ppls decided to completely ignore the work jboss put in to adding ajax to JSF[2]. I know that's the choice I would have made. 
5. xml is parsed and traversed in tree every request (ajax included). Today in 2015 we know that xml is bad because its slow. In fact, XML surpasses other data exchange formats in almost every other category: schema expressiveness, built in query language, reference types. But it's slow. XPATH is slow. Parsing on server is slow. Parsing on client is slow. Reading is slow. Writing is slow. Diffing is slow. 
6. because xml is slow, and jsf wasn't made for ajax, things are fragile. too many events and it dies. too special characters and it dies. too specific re-render instructions and it dies. two forms and on ie and it dies. 
7. Special characters: if you send valid html escape codes, richfaces server will parse them fine. If you send them back, they will display fine. If you a4j rerender them it will die. silently. unless you change your code, add an a4j:log, you can see xml parsing errors on the client. all browsers xml parsers are different in which characters they accept. 
8. WYSIWYG editor with custom css'd templates. oh man don't go there. Every browser treats keys differently (Enter is <br> or <p> or <div> depending on which browser you are using) Every browser treats whitespace differently. 400 line javascript monkey patch sitting on top of our editor for that right there. The 10x-law-of-demeter-violating-hack[5] discovered and written by me in 4 hours thank you very much. Debugged corner cases over the course of 3 weeks and I still can't get enters to work when cursor is at top level and no text has been written. I think richfaces just eats 'enter's.
9. Richfaces eats 'a's. You just can't do it. You are in a pick list and want to select all? you are in a pick list and you SIMPLY WANT TO SCROLL TO THE 'A's. Nope. Richfaces eats 'a's.
10. if you have one view root, and you want to talk to another view root, you can't. you encode your message in the url and make another request, creating another instance. Or you create a session scope or above bean that will serve as message communicator. Or you can hack javascript into giving you references to other views which give references to other richfaces objects. 
11. If you open up the js debugger, you will see an exception ONLY SOMETIMES that A4J.log is not defined. it is defined. Richfaces defined it everywhere. Twice. You defined it everywhere. Twice. js debugger will pause where the exception is thrown and it will be defined. but the exception doesn't go away. It's just something you have to ignore.
12. Lets say you have a custom page you wrote in a day. it's decently sized, couple hundred lines. For some reason, a4j requires up to 85MB allocations to process this. I don't care if you don't have backing beans, I don't care that you are not storing anything in session. a4j just allocates crap per session. Of course it's all garbage at the end of the request.  
13. What the above means is that each of your servers will have a hard and low limit of "in flight" requests it can process. This past week I've OOM'd[4] an old 32 bit jrockit weblogic 10.3 server instance with just 10 concurrent requests to a large richfaces 3 page. 
14. Richfaces 3 is ABANDONWARE [3] -- "old and unsupported" 
15.  Any javascript error while in an oncomplete= is invisible and swallowed.

welp. I forgot the rest of the lessons learned, which is why I'm going to start posting every week. 

[0] http://livedemo.exadel.com/richfaces-demo/

[1] http://blog.jquery.com/2009/01/14/jquery-1-3-released/

[2]https://developer.jboss.org/wiki/RichFacesMigrationGuide33x-4xMigration?_sscc=t

[3]https://developer.jboss.org/thread/241053

[4]OutOfMemoryException. see https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks002.html

[5] https://en.wikipedia.org/wiki/Law_of_Demeter

Your first discussion when you sit down to make a game.

First questions you want to make a game:

HCI
What are the Humans?
What are the Computers?
What is their Interaction?

Acceptable answers:
I'm going to make a game for the masses with millions dollars backing me on all modern consoles.
I'm going to make a game for my friends and Reddit to play on their computer.
I'm going to make a game for /b/ that they will jack off to on their tablets.
I'm going to make a game for Mrs. Chandler that shows how the expressive interaction of interpretative dance
can be integrated with the assistance of a mechanical third party to create something that is worthy to be
called literature.
I'm going to make a game for everyone on the internet that will incorporate drag and drop features in novel ways
that augment your brain with gluing gooey gluey goodness.
I'm going to make an in your face absurd machismo beat 'em up that contains many new additions to the field
such as automatic illustrating techniques that actually work.
I'm going to make a game on PC that makes you wish you were dead, or at least made you want to kill all those to
toture others.
I'm going to make a game that is like bill and teds excellent adventure but with more things to learn because learning is fun.
I'm going to make a game for runners/cyclists/gazelles that gives them another reality to be in while doing that exercise of their
choice. It would be like noby noby boy in it's social pointlessness.
I'm going to make a program for programmers to drink beer and program together.
I'm going to make a program for dnd nerds to drink and program together.
I'm going to make a beer proof keyboard that works well.
I'm going to make a touch corded keyboard.
I'm going to make a VIM for corded keyboards.
I'm going to make a homeless shelter that teaches people how to program. Program for food.
The programmers will need to complete lessons based on fingerprint? eye scan?
I'm going to make a map based games to end map based gaming.
I'm going to write a intuitive soundboard database (trees).
I'm going to write a game so passive it might as well be television. You control whatever little
you control with your eyes.
I'm going to write a game that makes political bullshit impossible}}}}.
I'm going to write something where working isn't interacting with the machine, but getting whatever you are doing done.
I'm going to write a game to create meaningful connections.
I'm going to write a game to make people feel love.
I'm going to write a game that will make the new culture seem obvious.

I'm going to create a pen that records where it's been written to. 3d map of the pen tip.

I'm going to create a touch corded keyboard on the steering wheel of the car.

I'm going to create a portal for thinking about your program : data flow? interface problems?

I'm going to create a game to make Mel love Vash more.

I'm going to make a game to make me feel less guilty about being white.

I'm going to make a game to help mathematicians think.

I want to create a game so that using your brain does not feel like work.


I'm going to make beer proof and homeless people proof keyboards.

I want to give my life to a cause.

I want to take poetry.

Poverty Part 3.

Completing a task with comrades that needs doing.

Well, the corporation fits that bill if you convince yourself your company needs to exist. (Non-Trivial task mind you)

The camaraderie comes from the difficulty of the task. People existing close to each other completing things are just going to become friends. If not then.. I don't know. Organizing friends is a difficult task. "Culture" corporations call it. You have to fit with the companies or you just won't get in the door.

Gangs I call it sometimes when I'm feeling down.

Step 1. Make prison's more mother fucking anti-social. Putting all those people together? Seriously? Colleges charge $5,000 per semester to do this to your god damn highschool kid (dorms) and we are giving the experience away for free to those who committed acts against society?

You are setting this up to say: your regular society sucks, but if you get inside you will be around people all the time. The fuck is that about?

Step 2. Turn these missions into a more social event. I see people lining up at the mission and I'm envious of the large groups of friends that interact while waiting for food. Of course, when I'm closer I see they mostly are angry at each other. It's hard to like other people while poor. This is why you need the other two elements: getting things done, and being needed.

Poverty. Part 2.

By the way, want to stop violence? Legalize.

Oliver Gospel Mission takes a Man on the Mountain point of view. The Cathedral.

I think Bazaar. I think Gas Stations.

Ponder for a second: what would life in this country be like if to operate a gas station, or to sell foods with saturated fat, or to sell cigarettes, you must operate a mission for the poor in the same building?

Approved missions:
1. Food Kitchen.
2. Drug rehabilitation clinic.
3. Job hunting office.
4. Volunteering corps office.
5. Beds.
6. Elderly Care Centers.
7. Veterinary Services(Seeing eye dogs get sick too).

You would design each approved mission in legalize so they would have roughly the same operational upkeep cost, and maybe even say you must give 10% of the station's profits to supporting this mission.

But we didn't take very much from the Army here. The task is to take the Army in mind as well, otherwise it just won't stick.

Poverty. Part 1.

Poverty. I want to solve it.

I want to seed some of your thoughts tonight with 2 things.

For the first seed I want to bring you to the south-east corner of Assembly and Taylor in Columbia South Carolina: the Oliver Gospel Mission.

http://www.olivergospelmission.org/

Your thought starts here because for 120 years they have been doing it right.

Welcome to the Oliver Gospel Mission. After 123 years, we are continuing our commitment to reaching out to the broken and homeless in Columbia. We have renewed the emphasis on equipping them to become spiritually fulfilled and functioning members of society who are making a positive impact on the quality of life in the Columbia area. For example, men who were previously unemployed are securing jobs, even in these tough economic times. Others are being reunited with estranged family members. Above all else, we continue to provide hope to people who have lost all sense of significance in a society that puts such a high value on material success. I am confident that as you browse our site you will be surprised by all we’re doing in the community, and find ways in which you can help. We encourage you to contact us for a tour. We’d love to have you visit the Mission and hope to hear from you soon!

-- President/CEO message

Re-read that message as if you were Hermione, look between the lines. Good progress is easy to find, as are ideological holes. Good Seed.

Second seed for this genetic experiment of a discussion we will find at 1225 Bluff Road, Columbia South Carolina. The National Guard.

Your thought must go here because since Neolithic times armies have been doing something right.

Now I know what you are thinking, It's the God Damn Army Bill. They fucking kill people. These are all murderers that deserve nothing more than death themselves. Every dollar spent on defense is a dollar stolen from someone's dinner. Etc.

But I've brought you here for a reason.

This one you really have to put on your Hermione glasses:

The Guard answers the call, at home or abroad.
Share

The incredible versatility of the National Guard enables its troops to respond to domestic emergencies, combat missions, counterdrug efforts, reconstruction missions and more—all with equal speed, strength and efficiency.

Whether the call is coming from the state governor or directly from the president of the United States, Guard Soldiers are always ready and always there.
You'll serve your community, your state and your country.

As a Guard Soldier, you can expect your primary area of operation to be your home state, following the leadership of your state adjutant general and governor. This may include community efforts, responding to wildfires or floods. Or, it may include serving overseas, training foreign forces.

You will also be prepared to mobilize when directed by the president. This may include overseas service or domestic, such as serving along the U.S.-Mexico border.

So, what does the National Guard do? Whatever is needed, wherever it is needed.

If you couldn't quite do it for that last one: I'll help. The first is the last line: "Whatever is needed, wherever it is needed." You are *needed* Charlie. We need you. I need you.

Doesn't that conjure up a fuzzy feeling... Dwell on that feeling for a little bit.

The second key is the rest of it. Describing "stuff to do." You'll be productive! Rob always described it as an itch he had.

Dwell on how you feel when you are getting things done contrasted with when you are not getting things done for a little bit.

The third key is entirely implied. You ask any soldier: Why did you fight? and they'll respond for my friends. For the people right next to me.

Community.

Not the community that people like to think of as their city. But real community. Real brotherhood. Camaraderie. Friendship.

That's it right there. You want to help mommy with the dishes? Do it. You are needed.