Ubuntu time

This week was a little all over the place.  I helped debug jpa queries(equals in jpql is case sensitive. Why would you expect it not to be???) I helped tighten up ssl on IIS6.(5 registry keys to create? Really? I hate that system config location is the same spot as app config locations. Linux is no better with /etc...) I wrote 126 java interfaces. (Daos for a very large web service). I wrote Ruby to spit out java for the osgi services that needed all those interfaces. I learned about gradle and wrote a dependency downloader script. I used that to compare cxf bundle (58 mb with a bunch of jars) and metro osgi (38mb with only a few jars.) Metro also looks more promising because its the jax ws reference implementation. I learned about dependendency resolution with scr's @Reference. I learned about enRoutes "compile-only" hack to make sure apis are pushed out with an implementation and never included in an osgi runtime alone. ( you have to add the api package to exports in bndtools view of your provider project's bnd.bnd and it will copy the class files. It's a bit too magic for my taste but whatever)

And I went in for 4 hours (3-7pm) Saturday so I could install Ubuntu on the server I bought. I'm a bit worried about weblogic... But it will force me to finish the bamboo deployments on check-in.... Hopefully it will force me to finish the weblogic spi jar to wrap our weblogic dependency as much as possible...

Also I need to write a basic telnet network stack for LVM. That is ULTRA dumb that you need to be physically at the server to enter the LUKS pw to decrypt at boot time. Like. WTF. I think I saw some solutions but they seem old.

When the boss skips town

This week my boss left town. A patch release went in but needed a patch release itself. That also went in.

We have an intern and I didn't quite know how much freedom to give. We deploy on WebLogic 10.3 so there are limitations with what one can build. Also my experience with that platform enables me to see issues beforehand.

Additionally, I came up with a deployment idea. The intern knew nothing of ears. I am also struggling to explain mocking, as I just read the tutorial myself.  The main algorithm of the app I came up with as well. I even just now writing this came up with an idea for a DISPLAY_WHEN column that is a velocity template to determine if a question is displayed or not.

In other news, I've been working with bndtools on Eclipse Mars. I've been trying to do it right: interfaces with javadoc and unit tests will be mockito like the intern is doing. We will see how much that slows things down.

When Spinning Plates Crash

JEE was not made for the working man. 

I mean, if you look at intro docs[0] you are going to see things like custom deployment descriptor capability. In practice what this means is that they expect a deployment person to take an existing working application, in a custom zip file, open the same file that is in charge of the whole container (how it is deployed, how things are named, what classes are and are not loaded) edit incredibly ill-commented xml files, and expect it to work in a new environment straight away. 

No. JEE was made by corporations with agendas. BEA, Oracle, IBM etc.. each with their own "value added" piece. Specifications written by corporations are nothing more than joint advertising efforts for their products.

An anonymous grey beard from the early cocaine-80's days of selling computer hardware told me once he went to a demo with a machine the size of a room that only had their tech lead inside with a small system to fake out the real thing. For $3 million dollars they wanted this thing. A guy in a box literally hard-wiring answers. 

I spent all week on getting each production version it's own building release branch. I'm still not done. 

• It's maven's SNAPSHOT. 
• It's svn. 
• It's ongoing poor deployment practices. 
• It's slow corp machines.
• It's corrupted outlook pst files.
• It's rushed integrations with no real transaction support.
• It's huge static custom css
• It's 100 degree heat walking to and from work
• It's interviews taking 90 minutes.
• It's bamboo's not latest branch version but current repository version.
• It's the COMPLETE LACK OF AUTOMATED TAPE BACKUPS.
• the list goes on and on and on and on

[0]: We are on jee5 still... so.. 

When You Bungle a Deployment

Ounce of Prevention

As senior reviewer of code, it is my responsibility to review all incoming changes for possible performance hits.

Obviously bad:
  session.addAttribute(new UUID(), new LargeXMLTree())
not so obviously bad (but essentially equivalent)

    < ... rendered="#{conditionforsomelargepage}">
      <a4j:include viewId="#{somelargepage}" />
    </ ... >

For new code, I made sure that the somelargepage was blank.xhtml if rendered was false. I didn't change the old pages. What I didn't realize (this release took 8 months from inception to get out the door, I forgot some changes alright?) was that old pages used to be included like

    < ... rendered="#{conditionforsomelargepage}">
      <ui:include src="#{somelargepage}">
    </ ..>

Which isn't bad at all. Unless you have ui:debug defined anywhere, in which case it is.

Pound of Cure

It's hard to tell your servers are supersaturated. The code worked fine before. We have not enabled new UI features yet. It never came up in test. The logs show different errors each time a server goes down. Servers are going down once every few hours. What happened?

We have no response time analytic capabilities. We have no baseline for time spent in gc. We have no way of telling session sizes in prod. We have no way of telling if this is a rogue action or a rising tide. We have no way of telling if this is a app memory bug, or a server configuration. We have no load testing capabilities.

Nightmare. We had hints though. The servers would be fine with no errors for a few hours until west coast people signed on. The logs would show nothing except for db timeouts of very basic queries. It happened around the same time each day. Eventually we rolled it back. Time wasted: 1 week.

Wasted Effort

But we got out of it. Idea one was to write a service to determine session size. *Heavily* modified jamm was the answer here.

This was a waste of time and did not contribute to final solution. Hey, I have a jsp that I can thread into a war now that will give you session size though. Not all bad. Time wasted: 1 week.

Repro Found

Step two was to write selenium tests from scratch. 1500 lines of code and one hijacked OCR server later, we were able to reproduce the same server behavior. Time spent: 1 week.

Using a trial version of YourKit, I spent 6 hours running the same selenium test case and taking snapshots. By the numbers: new code spent 156% the memory per session that the old code did. New code did not change app memory used (retained - session). I cannot stress how much this overall view was needed to prove that it wasn't a leak, or application memory bug.

Solution

After you can reproduce a problem, it becomes almost trivial to fix. You can literally just tweak at random until you see the problem go away.

Of course, I knew of an optimization beforehand, having it applied it to the new code. So the fix was not random.

Post Notes

I wrote many emails about how our testing infrastructure had holes that let this problem go though. I wrote many more emails about the money and time costs of fixing the infrastructure, and the pros and cons of different ways of fixing our software development life cycle.

Overall, it was one of the more interesting tasks I've had, as "proving" is in my DNA as a maths major.

Final lesson: large JSF apps simply do not scale.

The nothingness abyss referred to as Richfaces 3

This blog really didn't turn out to be about math and programming did it????

Well, let's change that. 

I'm going to talk now about richfaces 3.3.x. [0]

This is what you need if you are programming in richfaces 3: [0]

That's it. If that's down, you can't program in richfaces 3. 

Other quirks I've learned while using it for the past 3 years:

1. a4j has bad xml tree code. like, who hired that *** excuse for a *** *** * and ***?
2. If you are going to open a new browser window, and expect both windows to be used, put the code for the new window in a different war. 
3. richfaces programming style is jquery 1.3, but the thought process is pre-1.3. In case you forgot, or were still in diapers, jquery 1.3 was when they invented feature detection [1]. Thus trying to support any browsers that did not exist in 2009, you are going to have a Bad Time.
4. As far as I can tell JSF2 ppls decided to completely ignore the work jboss put in to adding ajax to JSF[2]. I know that's the choice I would have made. 
5. xml is parsed and traversed in tree every request (ajax included). Today in 2015 we know that xml is bad because its slow. In fact, XML surpasses other data exchange formats in almost every other category: schema expressiveness, built in query language, reference types. But it's slow. XPATH is slow. Parsing on server is slow. Parsing on client is slow. Reading is slow. Writing is slow. Diffing is slow. 
6. because xml is slow, and jsf wasn't made for ajax, things are fragile. too many events and it dies. too special characters and it dies. too specific re-render instructions and it dies. two forms and on ie and it dies. 
7. Special characters: if you send valid html escape codes, richfaces server will parse them fine. If you send them back, they will display fine. If you a4j rerender them it will die. silently. unless you change your code, add an a4j:log, you can see xml parsing errors on the client. all browsers xml parsers are different in which characters they accept. 
8. WYSIWYG editor with custom css'd templates. oh man don't go there. Every browser treats keys differently (Enter is <br> or <p> or <div> depending on which browser you are using) Every browser treats whitespace differently. 400 line javascript monkey patch sitting on top of our editor for that right there. The 10x-law-of-demeter-violating-hack[5] discovered and written by me in 4 hours thank you very much. Debugged corner cases over the course of 3 weeks and I still can't get enters to work when cursor is at top level and no text has been written. I think richfaces just eats 'enter's.
9. Richfaces eats 'a's. You just can't do it. You are in a pick list and want to select all? you are in a pick list and you SIMPLY WANT TO SCROLL TO THE 'A's. Nope. Richfaces eats 'a's.
10. if you have one view root, and you want to talk to another view root, you can't. you encode your message in the url and make another request, creating another instance. Or you create a session scope or above bean that will serve as message communicator. Or you can hack javascript into giving you references to other views which give references to other richfaces objects. 
11. If you open up the js debugger, you will see an exception ONLY SOMETIMES that A4J.log is not defined. it is defined. Richfaces defined it everywhere. Twice. You defined it everywhere. Twice. js debugger will pause where the exception is thrown and it will be defined. but the exception doesn't go away. It's just something you have to ignore.
12. Lets say you have a custom page you wrote in a day. it's decently sized, couple hundred lines. For some reason, a4j requires up to 85MB allocations to process this. I don't care if you don't have backing beans, I don't care that you are not storing anything in session. a4j just allocates crap per session. Of course it's all garbage at the end of the request.  
13. What the above means is that each of your servers will have a hard and low limit of "in flight" requests it can process. This past week I've OOM'd[4] an old 32 bit jrockit weblogic 10.3 server instance with just 10 concurrent requests to a large richfaces 3 page. 
14. Richfaces 3 is ABANDONWARE [3] -- "old and unsupported" 
15.  Any javascript error while in an oncomplete= is invisible and swallowed.

welp. I forgot the rest of the lessons learned, which is why I'm going to start posting every week. 

[0] http://livedemo.exadel.com/richfaces-demo/

[1] http://blog.jquery.com/2009/01/14/jquery-1-3-released/

[2]https://developer.jboss.org/wiki/RichFacesMigrationGuide33x-4xMigration?_sscc=t

[3]https://developer.jboss.org/thread/241053

[4]OutOfMemoryException. see https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks002.html

[5] https://en.wikipedia.org/wiki/Law_of_Demeter

Your first discussion when you sit down to make a game.

First questions you want to make a game:

HCI
What are the Humans?
What are the Computers?
What is their Interaction?

Acceptable answers:
I'm going to make a game for the masses with millions dollars backing me on all modern consoles.
I'm going to make a game for my friends and Reddit to play on their computer.
I'm going to make a game for /b/ that they will jack off to on their tablets.
I'm going to make a game for Mrs. Chandler that shows how the expressive interaction of interpretative dance
can be integrated with the assistance of a mechanical third party to create something that is worthy to be
called literature.
I'm going to make a game for everyone on the internet that will incorporate drag and drop features in novel ways
that augment your brain with gluing gooey gluey goodness.
I'm going to make an in your face absurd machismo beat 'em up that contains many new additions to the field
such as automatic illustrating techniques that actually work.
I'm going to make a game on PC that makes you wish you were dead, or at least made you want to kill all those to
toture others.
I'm going to make a game that is like bill and teds excellent adventure but with more things to learn because learning is fun.
I'm going to make a game for runners/cyclists/gazelles that gives them another reality to be in while doing that exercise of their
choice. It would be like noby noby boy in it's social pointlessness.
I'm going to make a program for programmers to drink beer and program together.
I'm going to make a program for dnd nerds to drink and program together.
I'm going to make a beer proof keyboard that works well.
I'm going to make a touch corded keyboard.
I'm going to make a VIM for corded keyboards.
I'm going to make a homeless shelter that teaches people how to program. Program for food.
The programmers will need to complete lessons based on fingerprint? eye scan?
I'm going to make a map based games to end map based gaming.
I'm going to write a intuitive soundboard database (trees).
I'm going to write a game so passive it might as well be television. You control whatever little
you control with your eyes.
I'm going to write a game that makes political bullshit impossible}}}}.
I'm going to write something where working isn't interacting with the machine, but getting whatever you are doing done.
I'm going to write a game to create meaningful connections.
I'm going to write a game to make people feel love.
I'm going to write a game that will make the new culture seem obvious.

I'm going to create a pen that records where it's been written to. 3d map of the pen tip.

I'm going to create a touch corded keyboard on the steering wheel of the car.

I'm going to create a portal for thinking about your program : data flow? interface problems?

I'm going to create a game to make Mel love Vash more.

I'm going to make a game to make me feel less guilty about being white.

I'm going to make a game to help mathematicians think.

I want to create a game so that using your brain does not feel like work.


I'm going to make beer proof and homeless people proof keyboards.

I want to give my life to a cause.

I want to take poetry.

Poverty Part 3.

Completing a task with comrades that needs doing.

Well, the corporation fits that bill if you convince yourself your company needs to exist. (Non-Trivial task mind you)

The camaraderie comes from the difficulty of the task. People existing close to each other completing things are just going to become friends. If not then.. I don't know. Organizing friends is a difficult task. "Culture" corporations call it. You have to fit with the companies or you just won't get in the door.

Gangs I call it sometimes when I'm feeling down.

Step 1. Make prison's more mother fucking anti-social. Putting all those people together? Seriously? Colleges charge $5,000 per semester to do this to your god damn highschool kid (dorms) and we are giving the experience away for free to those who committed acts against society?

You are setting this up to say: your regular society sucks, but if you get inside you will be around people all the time. The fuck is that about?

Step 2. Turn these missions into a more social event. I see people lining up at the mission and I'm envious of the large groups of friends that interact while waiting for food. Of course, when I'm closer I see they mostly are angry at each other. It's hard to like other people while poor. This is why you need the other two elements: getting things done, and being needed.