When You Bungle a Deployment

Ounce of Prevention

As senior reviewer of code, it is my responsibility to review all incoming changes for possible performance hits.

Obviously bad:
  session.addAttribute(new UUID(), new LargeXMLTree())
not so obviously bad (but essentially equivalent)

    < ... rendered="#{conditionforsomelargepage}">
      <a4j:include viewId="#{somelargepage}" />
    </ ... >

For new code, I made sure that the somelargepage was blank.xhtml if rendered was false. I didn't change the old pages. What I didn't realize (this release took 8 months from inception to get out the door, I forgot some changes alright?) was that old pages used to be included like

    < ... rendered="#{conditionforsomelargepage}">
      <ui:include src="#{somelargepage}">
    </ ..>

Which isn't bad at all. Unless you have ui:debug defined anywhere, in which case it is.

Pound of Cure

It's hard to tell your servers are supersaturated. The code worked fine before. We have not enabled new UI features yet. It never came up in test. The logs show different errors each time a server goes down. Servers are going down once every few hours. What happened?

We have no response time analytic capabilities. We have no baseline for time spent in gc. We have no way of telling session sizes in prod. We have no way of telling if this is a rogue action or a rising tide. We have no way of telling if this is a app memory bug, or a server configuration. We have no load testing capabilities.

Nightmare. We had hints though. The servers would be fine with no errors for a few hours until west coast people signed on. The logs would show nothing except for db timeouts of very basic queries. It happened around the same time each day. Eventually we rolled it back. Time wasted: 1 week.

Wasted Effort

But we got out of it. Idea one was to write a service to determine session size. *Heavily* modified jamm was the answer here.

This was a waste of time and did not contribute to final solution. Hey, I have a jsp that I can thread into a war now that will give you session size though. Not all bad. Time wasted: 1 week.

Repro Found

Step two was to write selenium tests from scratch. 1500 lines of code and one hijacked OCR server later, we were able to reproduce the same server behavior. Time spent: 1 week.

Using a trial version of YourKit, I spent 6 hours running the same selenium test case and taking snapshots. By the numbers: new code spent 156% the memory per session that the old code did. New code did not change app memory used (retained - session). I cannot stress how much this overall view was needed to prove that it wasn't a leak, or application memory bug.

Solution

After you can reproduce a problem, it becomes almost trivial to fix. You can literally just tweak at random until you see the problem go away.

Of course, I knew of an optimization beforehand, having it applied it to the new code. So the fix was not random.

Post Notes

I wrote many emails about how our testing infrastructure had holes that let this problem go though. I wrote many more emails about the money and time costs of fixing the infrastructure, and the pros and cons of different ways of fixing our software development life cycle.

Overall, it was one of the more interesting tasks I've had, as "proving" is in my DNA as a maths major.

Final lesson: large JSF apps simply do not scale.

The nothingness abyss referred to as Richfaces 3

This blog really didn't turn out to be about math and programming did it????

Well, let's change that. 

I'm going to talk now about richfaces 3.3.x. [0]

This is what you need if you are programming in richfaces 3: [0]

That's it. If that's down, you can't program in richfaces 3. 

Other quirks I've learned while using it for the past 3 years:

1. a4j has bad xml tree code. like, who hired that *** excuse for a *** *** * and ***?
2. If you are going to open a new browser window, and expect both windows to be used, put the code for the new window in a different war. 
3. richfaces programming style is jquery 1.3, but the thought process is pre-1.3. In case you forgot, or were still in diapers, jquery 1.3 was when they invented feature detection [1]. Thus trying to support any browsers that did not exist in 2009, you are going to have a Bad Time.
4. As far as I can tell JSF2 ppls decided to completely ignore the work jboss put in to adding ajax to JSF[2]. I know that's the choice I would have made. 
5. xml is parsed and traversed in tree every request (ajax included). Today in 2015 we know that xml is bad because its slow. In fact, XML surpasses other data exchange formats in almost every other category: schema expressiveness, built in query language, reference types. But it's slow. XPATH is slow. Parsing on server is slow. Parsing on client is slow. Reading is slow. Writing is slow. Diffing is slow. 
6. because xml is slow, and jsf wasn't made for ajax, things are fragile. too many events and it dies. too special characters and it dies. too specific re-render instructions and it dies. two forms and on ie and it dies. 
7. Special characters: if you send valid html escape codes, richfaces server will parse them fine. If you send them back, they will display fine. If you a4j rerender them it will die. silently. unless you change your code, add an a4j:log, you can see xml parsing errors on the client. all browsers xml parsers are different in which characters they accept. 
8. WYSIWYG editor with custom css'd templates. oh man don't go there. Every browser treats keys differently (Enter is <br> or <p> or <div> depending on which browser you are using) Every browser treats whitespace differently. 400 line javascript monkey patch sitting on top of our editor for that right there. The 10x-law-of-demeter-violating-hack[5] discovered and written by me in 4 hours thank you very much. Debugged corner cases over the course of 3 weeks and I still can't get enters to work when cursor is at top level and no text has been written. I think richfaces just eats 'enter's.
9. Richfaces eats 'a's. You just can't do it. You are in a pick list and want to select all? you are in a pick list and you SIMPLY WANT TO SCROLL TO THE 'A's. Nope. Richfaces eats 'a's.
10. if you have one view root, and you want to talk to another view root, you can't. you encode your message in the url and make another request, creating another instance. Or you create a session scope or above bean that will serve as message communicator. Or you can hack javascript into giving you references to other views which give references to other richfaces objects. 
11. If you open up the js debugger, you will see an exception ONLY SOMETIMES that A4J.log is not defined. it is defined. Richfaces defined it everywhere. Twice. You defined it everywhere. Twice. js debugger will pause where the exception is thrown and it will be defined. but the exception doesn't go away. It's just something you have to ignore.
12. Lets say you have a custom page you wrote in a day. it's decently sized, couple hundred lines. For some reason, a4j requires up to 85MB allocations to process this. I don't care if you don't have backing beans, I don't care that you are not storing anything in session. a4j just allocates crap per session. Of course it's all garbage at the end of the request.  
13. What the above means is that each of your servers will have a hard and low limit of "in flight" requests it can process. This past week I've OOM'd[4] an old 32 bit jrockit weblogic 10.3 server instance with just 10 concurrent requests to a large richfaces 3 page. 
14. Richfaces 3 is ABANDONWARE [3] -- "old and unsupported" 
15.  Any javascript error while in an oncomplete= is invisible and swallowed.

welp. I forgot the rest of the lessons learned, which is why I'm going to start posting every week. 

[0] http://livedemo.exadel.com/richfaces-demo/

[1] http://blog.jquery.com/2009/01/14/jquery-1-3-released/

[2]https://developer.jboss.org/wiki/RichFacesMigrationGuide33x-4xMigration?_sscc=t

[3]https://developer.jboss.org/thread/241053

[4]OutOfMemoryException. see https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks002.html

[5] https://en.wikipedia.org/wiki/Law_of_Demeter

Your first discussion when you sit down to make a game.

First questions you want to make a game:

HCI
What are the Humans?
What are the Computers?
What is their Interaction?

Acceptable answers:
I'm going to make a game for the masses with millions dollars backing me on all modern consoles.
I'm going to make a game for my friends and Reddit to play on their computer.
I'm going to make a game for /b/ that they will jack off to on their tablets.
I'm going to make a game for Mrs. Chandler that shows how the expressive interaction of interpretative dance
can be integrated with the assistance of a mechanical third party to create something that is worthy to be
called literature.
I'm going to make a game for everyone on the internet that will incorporate drag and drop features in novel ways
that augment your brain with gluing gooey gluey goodness.
I'm going to make an in your face absurd machismo beat 'em up that contains many new additions to the field
such as automatic illustrating techniques that actually work.
I'm going to make a game on PC that makes you wish you were dead, or at least made you want to kill all those to
toture others.
I'm going to make a game that is like bill and teds excellent adventure but with more things to learn because learning is fun.
I'm going to make a game for runners/cyclists/gazelles that gives them another reality to be in while doing that exercise of their
choice. It would be like noby noby boy in it's social pointlessness.
I'm going to make a program for programmers to drink beer and program together.
I'm going to make a program for dnd nerds to drink and program together.
I'm going to make a beer proof keyboard that works well.
I'm going to make a touch corded keyboard.
I'm going to make a VIM for corded keyboards.
I'm going to make a homeless shelter that teaches people how to program. Program for food.
The programmers will need to complete lessons based on fingerprint? eye scan?
I'm going to make a map based games to end map based gaming.
I'm going to write a intuitive soundboard database (trees).
I'm going to write a game so passive it might as well be television. You control whatever little
you control with your eyes.
I'm going to write a game that makes political bullshit impossible}}}}.
I'm going to write something where working isn't interacting with the machine, but getting whatever you are doing done.
I'm going to write a game to create meaningful connections.
I'm going to write a game to make people feel love.
I'm going to write a game that will make the new culture seem obvious.

I'm going to create a pen that records where it's been written to. 3d map of the pen tip.

I'm going to create a touch corded keyboard on the steering wheel of the car.

I'm going to create a portal for thinking about your program : data flow? interface problems?

I'm going to create a game to make Mel love Vash more.

I'm going to make a game to make me feel less guilty about being white.

I'm going to make a game to help mathematicians think.

I want to create a game so that using your brain does not feel like work.


I'm going to make beer proof and homeless people proof keyboards.

I want to give my life to a cause.

I want to take poetry.

Poverty Part 3.

Completing a task with comrades that needs doing.

Well, the corporation fits that bill if you convince yourself your company needs to exist. (Non-Trivial task mind you)

The camaraderie comes from the difficulty of the task. People existing close to each other completing things are just going to become friends. If not then.. I don't know. Organizing friends is a difficult task. "Culture" corporations call it. You have to fit with the companies or you just won't get in the door.

Gangs I call it sometimes when I'm feeling down.

Step 1. Make prison's more mother fucking anti-social. Putting all those people together? Seriously? Colleges charge $5,000 per semester to do this to your god damn highschool kid (dorms) and we are giving the experience away for free to those who committed acts against society?

You are setting this up to say: your regular society sucks, but if you get inside you will be around people all the time. The fuck is that about?

Step 2. Turn these missions into a more social event. I see people lining up at the mission and I'm envious of the large groups of friends that interact while waiting for food. Of course, when I'm closer I see they mostly are angry at each other. It's hard to like other people while poor. This is why you need the other two elements: getting things done, and being needed.

Poverty. Part 2.

By the way, want to stop violence? Legalize.

Oliver Gospel Mission takes a Man on the Mountain point of view. The Cathedral.

I think Bazaar. I think Gas Stations.

Ponder for a second: what would life in this country be like if to operate a gas station, or to sell foods with saturated fat, or to sell cigarettes, you must operate a mission for the poor in the same building?

Approved missions:
1. Food Kitchen.
2. Drug rehabilitation clinic.
3. Job hunting office.
4. Volunteering corps office.
5. Beds.
6. Elderly Care Centers.
7. Veterinary Services(Seeing eye dogs get sick too).

You would design each approved mission in legalize so they would have roughly the same operational upkeep cost, and maybe even say you must give 10% of the station's profits to supporting this mission.

But we didn't take very much from the Army here. The task is to take the Army in mind as well, otherwise it just won't stick.

Poverty. Part 1.

Poverty. I want to solve it.

I want to seed some of your thoughts tonight with 2 things.

For the first seed I want to bring you to the south-east corner of Assembly and Taylor in Columbia South Carolina: the Oliver Gospel Mission.

http://www.olivergospelmission.org/

Your thought starts here because for 120 years they have been doing it right.

Welcome to the Oliver Gospel Mission. After 123 years, we are continuing our commitment to reaching out to the broken and homeless in Columbia. We have renewed the emphasis on equipping them to become spiritually fulfilled and functioning members of society who are making a positive impact on the quality of life in the Columbia area. For example, men who were previously unemployed are securing jobs, even in these tough economic times. Others are being reunited with estranged family members. Above all else, we continue to provide hope to people who have lost all sense of significance in a society that puts such a high value on material success. I am confident that as you browse our site you will be surprised by all we’re doing in the community, and find ways in which you can help. We encourage you to contact us for a tour. We’d love to have you visit the Mission and hope to hear from you soon!

-- President/CEO message

Re-read that message as if you were Hermione, look between the lines. Good progress is easy to find, as are ideological holes. Good Seed.

Second seed for this genetic experiment of a discussion we will find at 1225 Bluff Road, Columbia South Carolina. The National Guard.

Your thought must go here because since Neolithic times armies have been doing something right.

Now I know what you are thinking, It's the God Damn Army Bill. They fucking kill people. These are all murderers that deserve nothing more than death themselves. Every dollar spent on defense is a dollar stolen from someone's dinner. Etc.

But I've brought you here for a reason.

This one you really have to put on your Hermione glasses:

The Guard answers the call, at home or abroad.
Share

The incredible versatility of the National Guard enables its troops to respond to domestic emergencies, combat missions, counterdrug efforts, reconstruction missions and more—all with equal speed, strength and efficiency.

Whether the call is coming from the state governor or directly from the president of the United States, Guard Soldiers are always ready and always there.
You'll serve your community, your state and your country.

As a Guard Soldier, you can expect your primary area of operation to be your home state, following the leadership of your state adjutant general and governor. This may include community efforts, responding to wildfires or floods. Or, it may include serving overseas, training foreign forces.

You will also be prepared to mobilize when directed by the president. This may include overseas service or domestic, such as serving along the U.S.-Mexico border.

So, what does the National Guard do? Whatever is needed, wherever it is needed.

If you couldn't quite do it for that last one: I'll help. The first is the last line: "Whatever is needed, wherever it is needed." You are *needed* Charlie. We need you. I need you.

Doesn't that conjure up a fuzzy feeling... Dwell on that feeling for a little bit.

The second key is the rest of it. Describing "stuff to do." You'll be productive! Rob always described it as an itch he had.

Dwell on how you feel when you are getting things done contrasted with when you are not getting things done for a little bit.

The third key is entirely implied. You ask any soldier: Why did you fight? and they'll respond for my friends. For the people right next to me.

Community.

Not the community that people like to think of as their city. But real community. Real brotherhood. Camaraderie. Friendship.

That's it right there. You want to help mommy with the dishes? Do it. You are needed.

This one is bill.

What I think Stewart is advocating for (or maybe this just just what I would advocate for so I'm projecting) is that:
We advocate with honesty and sincerity. Don't misrepresent the truth to suit your own viewpoints and don't pretend to be a ludicrous parody of your own beliefs in order to force the discussion in your direction. Ask for what you really want. If you want an end to partial-birth abortion don't say you think all family planners are murders and should be on death row. If you want better financial regulation don't say you want bankers kicked out of their homes so that victims of fraudulent foreclosure can live in them.
We treat our opponents fairly. Avoid ad hominem attacks when they aren't relevant to the topic at hand (the almost never are).
We seek to maintain a productive dialog. Is your discuss convincing your opponent ? Is it convincing an undecided observer ? Is it rousing those in agreement with you to take productive action ? If not then you're wasting your breath and creating meaningless polarization.
We try to understand an opponent's perspective. Do I think our involvement in two wars over the course of a decade has turned out to be foolhardy and ill-advised--yes. Do I think many of the acts we've committed were probably criminal--I do. Do I think ulterior motives were involved--it's very likely. However, I don't honestly believe that anyone in power went home and told their SO, "Baby, I think we've found a way to make money by killing Ahab." Relying on dark and vile imagery to paint our opponents as bad people is intellectually shallow. Most people are not truly evil and even those who are could be prevented from doing bad things if we do what's needed to solve the elementary systemic flaws in our political institutions.
None of those points require advocating a position any more moderate than the one you honestly hold. It simply requires that you seek it in a way that's honest and productive and is more interested in moving the dialog forward than in "winning".

As a mathematics graduate student in the boonies, I'm going to counter point here on behalf of Andrew Jackson's mobocracy. I don't have time to read the Bush books. I simply do not have time enough to keep up on being literate when it comes to any topics The Daily Show deems important.

In college, we talked about how to "read texts." Experiment with which perspective you take when you read new ideas. Look for things missing, look for hidden discriminations.

However, modern speech writers took such a large shit in my mouth that I can't do that logically. It's impossible. I literally do not have enough processing power to consider everyone's problems. In New York City a few weeks ago I had this carnivale experience where ambulances were running and all the hundreds conversations on the street kept going. This was right down the street, someone was obviously hurt but these people have been convinced that this is not an issue for them; wall street is, Wisconsin is, Conservatives are, their job is.

Everyone has a voice, no one has, or is getting, a heart.