Vaadin

I worked with Vaadin this week. Trying to nail down form items.

If it's generic, it will be avoided. If it's too specific, it goes too slow, if it's too optimized it will take away too slow to make.

We all just want a framework that just works. Fast to write, secure, future proof, performant.

Aspirations

For every dream there is a reality check, but for every action there is an impetus. 

When OSGI-on-glassfish was the new hotness at my company, I spent 2 weeks trying to figure out how to get Smooks and Camel to load a flat file though EJBs. I rewrote it in 5 hours on Friday with SQL*Loader including an hour for load time.

Last week I spent 3 hours helping get a solid drools deployment done. I was only tertiary help. The code was written, but the deployment was just not there. If only they were just pulled from a DB....

I spent the rest of my time on Gatling. I am really new when it comes to Scala and I went down a lot of dead ends trying to load test a complex Vaadin app. I think I have an approach now but I'm still working on creating realistic workloads. 2 weeks and all I can do is click a button.

I just wish Vaadin came with a simple tool like SQL loader.

Blogger for Android sucks. It deleted text and now won't type. I finished this with something about golang being simple.
https://splice.com/blog/golang-improved-simplicity-reduced-maintenance/

Making peace with interfaces

While co-designing a new product, I had to also switch schools of Java.  

I came from an immutable-is-king school where all fields are public finals, constructors are private, and builders are fluent. In this school if you have a task you break it up into necessary data transfer objects. Any gateway communication like sql or soap is hand crafted and hidden inside methods of your data model.

I'm switching to a functional grouping of modules. In this way interfaces are designed while tests are created. Your tasks first get APIs, then get necessary data accessors. Public final fields are from this view too restrictive in data location.

For example in the first way a documentID is stored and a method to send a document then is in charge of using it.

In the second way, a document accessor interface is defined and an implementation is passed when any user needs a document, allowing any module to locate documents reliably and without duplication.

It makes pragmatic thinking more difficult while not limiting it. For this cost you get consistent reliable implementation visibility control.

More business and less programming if you ask me.

Re-implement would make the same mistakes

When you switch technologies you understand why things are the way they are. 

From pure Maven JEE5 we switched to "bndtools". bndtools is an eclipse plugin. bndtools is a way to rapidly work on bnd based projects. bnd is a rapid way to work on OSGI bundles. OSGI is a way to modulate Java code. A bundle is the OSGI module. 

QueryDSL is an API for database communication from Java. QueryDSL uses the pattern of creating a customized API based on your data model. It does this by generating Java code. I refer to this generated code as "Q classes" because they all start with Q.

The Maven way of generating these Q classes is with a plugin. Compiling and other things like testing and packaging is referred to as building. Maven has a build lifecycle which includes the normal phases like compile and test. One of the standard Maven lifecycle phases is "generate-sources". This phase is meant to cover exactly the use case QueryDSL uses, that is to generate Java used to code against. 

For the bndtools project my tech lead suggested eclipse's built in "builder" to call QueryDSL to generate the Q classes. I've never done that, took me a day or two of spare time. 

Of course this means the build box can't generate the Q classes. So now I have to it it in a Gradle task and call that from the eclipse builder. 

I miss Maven right now.

Bean mapping

I taught again this week. I don't know why o do it. Probably because I feel productive.

I worked on bean mapping. Having a layer protecting your exposure to third party helps... but at what cost?

I'm building that layer now for our intf with WebLogic. I don't quite understand why it was used in certain places but such is life.

Building it after is in that case of: it works, why spend time on it????? And the answer is technical debt. Startup costs, security costs.

People try to tack on lower cost of future maintenance but I don't buy it. If your car works, and someone offers you a service saying: "I'll replace your air intake manifold for $1,000, it will save you wear and tear on you engine" I'd say, "Hey. There were smart people testing and designing this originally, where is your proof yours is better?"

So we get layers. The most important thing in our program is our capability of changing out our ORM if you looked at lines of code.

The things I want to change are to go stackless; use akka, or vertx. Where is that bean mapping capability?

Web layers tell you to pollute your beans with json, request args, and post method mapping. DB layers tell you to pollute your beans with column, table, and join information. Service layers tell you to pollute your bean with validation information and XML information. So we end up with bean mappers.

I like raw java mappers (static converter utils) for the task.  Orika, Dozer, w\e just add bad, half implemented DSLs to learn the edge cases for. The java converter is easily updated and doesn't require anything from either layer other than setters.

Ubuntu time

This week was a little all over the place.  I helped debug jpa queries(equals in jpql is case sensitive. Why would you expect it not to be???) I helped tighten up ssl on IIS6.(5 registry keys to create? Really? I hate that system config location is the same spot as app config locations. Linux is no better with /etc...) I wrote 126 java interfaces. (Daos for a very large web service). I wrote Ruby to spit out java for the osgi services that needed all those interfaces. I learned about gradle and wrote a dependency downloader script. I used that to compare cxf bundle (58 mb with a bunch of jars) and metro osgi (38mb with only a few jars.) Metro also looks more promising because its the jax ws reference implementation. I learned about dependendency resolution with scr's @Reference. I learned about enRoutes "compile-only" hack to make sure apis are pushed out with an implementation and never included in an osgi runtime alone. ( you have to add the api package to exports in bndtools view of your provider project's bnd.bnd and it will copy the class files. It's a bit too magic for my taste but whatever)

And I went in for 4 hours (3-7pm) Saturday so I could install Ubuntu on the server I bought. I'm a bit worried about weblogic... But it will force me to finish the bamboo deployments on check-in.... Hopefully it will force me to finish the weblogic spi jar to wrap our weblogic dependency as much as possible...

Also I need to write a basic telnet network stack for LVM. That is ULTRA dumb that you need to be physically at the server to enter the LUKS pw to decrypt at boot time. Like. WTF. I think I saw some solutions but they seem old.

When the boss skips town

This week my boss left town. A patch release went in but needed a patch release itself. That also went in.

We have an intern and I didn't quite know how much freedom to give. We deploy on WebLogic 10.3 so there are limitations with what one can build. Also my experience with that platform enables me to see issues beforehand.

Additionally, I came up with a deployment idea. The intern knew nothing of ears. I am also struggling to explain mocking, as I just read the tutorial myself.  The main algorithm of the app I came up with as well. I even just now writing this came up with an idea for a DISPLAY_WHEN column that is a velocity template to determine if a question is displayed or not.

In other news, I've been working with bndtools on Eclipse Mars. I've been trying to do it right: interfaces with javadoc and unit tests will be mockito like the intern is doing. We will see how much that slows things down.

When Spinning Plates Crash

JEE was not made for the working man. 

I mean, if you look at intro docs[0] you are going to see things like custom deployment descriptor capability. In practice what this means is that they expect a deployment person to take an existing working application, in a custom zip file, open the same file that is in charge of the whole container (how it is deployed, how things are named, what classes are and are not loaded) edit incredibly ill-commented xml files, and expect it to work in a new environment straight away. 

No. JEE was made by corporations with agendas. BEA, Oracle, IBM etc.. each with their own "value added" piece. Specifications written by corporations are nothing more than joint advertising efforts for their products.

An anonymous grey beard from the early cocaine-80's days of selling computer hardware told me once he went to a demo with a machine the size of a room that only had their tech lead inside with a small system to fake out the real thing. For $3 million dollars they wanted this thing. A guy in a box literally hard-wiring answers. 

I spent all week on getting each production version it's own building release branch. I'm still not done. 

• It's maven's SNAPSHOT. 
• It's svn. 
• It's ongoing poor deployment practices. 
• It's slow corp machines.
• It's corrupted outlook pst files.
• It's rushed integrations with no real transaction support.
• It's huge static custom css
• It's 100 degree heat walking to and from work
• It's interviews taking 90 minutes.
• It's bamboo's not latest branch version but current repository version.
• It's the COMPLETE LACK OF AUTOMATED TAPE BACKUPS.
• the list goes on and on and on and on

[0]: We are on jee5 still... so.. 

When You Bungle a Deployment

Ounce of Prevention

As senior reviewer of code, it is my responsibility to review all incoming changes for possible performance hits.

Obviously bad:
  session.addAttribute(new UUID(), new LargeXMLTree())
not so obviously bad (but essentially equivalent)

    < ... rendered="#{conditionforsomelargepage}">
      <a4j:include viewId="#{somelargepage}" />
    </ ... >

For new code, I made sure that the somelargepage was blank.xhtml if rendered was false. I didn't change the old pages. What I didn't realize (this release took 8 months from inception to get out the door, I forgot some changes alright?) was that old pages used to be included like

    < ... rendered="#{conditionforsomelargepage}">
      <ui:include src="#{somelargepage}">
    </ ..>

Which isn't bad at all. Unless you have ui:debug defined anywhere, in which case it is.

Pound of Cure

It's hard to tell your servers are supersaturated. The code worked fine before. We have not enabled new UI features yet. It never came up in test. The logs show different errors each time a server goes down. Servers are going down once every few hours. What happened?

We have no response time analytic capabilities. We have no baseline for time spent in gc. We have no way of telling session sizes in prod. We have no way of telling if this is a rogue action or a rising tide. We have no way of telling if this is a app memory bug, or a server configuration. We have no load testing capabilities.

Nightmare. We had hints though. The servers would be fine with no errors for a few hours until west coast people signed on. The logs would show nothing except for db timeouts of very basic queries. It happened around the same time each day. Eventually we rolled it back. Time wasted: 1 week.

Wasted Effort

But we got out of it. Idea one was to write a service to determine session size. *Heavily* modified jamm was the answer here.

This was a waste of time and did not contribute to final solution. Hey, I have a jsp that I can thread into a war now that will give you session size though. Not all bad. Time wasted: 1 week.

Repro Found

Step two was to write selenium tests from scratch. 1500 lines of code and one hijacked OCR server later, we were able to reproduce the same server behavior. Time spent: 1 week.

Using a trial version of YourKit, I spent 6 hours running the same selenium test case and taking snapshots. By the numbers: new code spent 156% the memory per session that the old code did. New code did not change app memory used (retained - session). I cannot stress how much this overall view was needed to prove that it wasn't a leak, or application memory bug.

Solution

After you can reproduce a problem, it becomes almost trivial to fix. You can literally just tweak at random until you see the problem go away.

Of course, I knew of an optimization beforehand, having it applied it to the new code. So the fix was not random.

Post Notes

I wrote many emails about how our testing infrastructure had holes that let this problem go though. I wrote many more emails about the money and time costs of fixing the infrastructure, and the pros and cons of different ways of fixing our software development life cycle.

Overall, it was one of the more interesting tasks I've had, as "proving" is in my DNA as a maths major.

Final lesson: large JSF apps simply do not scale.

The nothingness abyss referred to as Richfaces 3

This blog really didn't turn out to be about math and programming did it????

Well, let's change that. 

I'm going to talk now about richfaces 3.3.x. [0]

This is what you need if you are programming in richfaces 3: [0]

That's it. If that's down, you can't program in richfaces 3. 

Other quirks I've learned while using it for the past 3 years:

1. a4j has bad xml tree code. like, who hired that *** excuse for a *** *** * and ***?
2. If you are going to open a new browser window, and expect both windows to be used, put the code for the new window in a different war. 
3. richfaces programming style is jquery 1.3, but the thought process is pre-1.3. In case you forgot, or were still in diapers, jquery 1.3 was when they invented feature detection [1]. Thus trying to support any browsers that did not exist in 2009, you are going to have a Bad Time.
4. As far as I can tell JSF2 ppls decided to completely ignore the work jboss put in to adding ajax to JSF[2]. I know that's the choice I would have made. 
5. xml is parsed and traversed in tree every request (ajax included). Today in 2015 we know that xml is bad because its slow. In fact, XML surpasses other data exchange formats in almost every other category: schema expressiveness, built in query language, reference types. But it's slow. XPATH is slow. Parsing on server is slow. Parsing on client is slow. Reading is slow. Writing is slow. Diffing is slow. 
6. because xml is slow, and jsf wasn't made for ajax, things are fragile. too many events and it dies. too special characters and it dies. too specific re-render instructions and it dies. two forms and on ie and it dies. 
7. Special characters: if you send valid html escape codes, richfaces server will parse them fine. If you send them back, they will display fine. If you a4j rerender them it will die. silently. unless you change your code, add an a4j:log, you can see xml parsing errors on the client. all browsers xml parsers are different in which characters they accept. 
8. WYSIWYG editor with custom css'd templates. oh man don't go there. Every browser treats keys differently (Enter is <br> or <p> or <div> depending on which browser you are using) Every browser treats whitespace differently. 400 line javascript monkey patch sitting on top of our editor for that right there. The 10x-law-of-demeter-violating-hack[5] discovered and written by me in 4 hours thank you very much. Debugged corner cases over the course of 3 weeks and I still can't get enters to work when cursor is at top level and no text has been written. I think richfaces just eats 'enter's.
9. Richfaces eats 'a's. You just can't do it. You are in a pick list and want to select all? you are in a pick list and you SIMPLY WANT TO SCROLL TO THE 'A's. Nope. Richfaces eats 'a's.
10. if you have one view root, and you want to talk to another view root, you can't. you encode your message in the url and make another request, creating another instance. Or you create a session scope or above bean that will serve as message communicator. Or you can hack javascript into giving you references to other views which give references to other richfaces objects. 
11. If you open up the js debugger, you will see an exception ONLY SOMETIMES that A4J.log is not defined. it is defined. Richfaces defined it everywhere. Twice. You defined it everywhere. Twice. js debugger will pause where the exception is thrown and it will be defined. but the exception doesn't go away. It's just something you have to ignore.
12. Lets say you have a custom page you wrote in a day. it's decently sized, couple hundred lines. For some reason, a4j requires up to 85MB allocations to process this. I don't care if you don't have backing beans, I don't care that you are not storing anything in session. a4j just allocates crap per session. Of course it's all garbage at the end of the request.  
13. What the above means is that each of your servers will have a hard and low limit of "in flight" requests it can process. This past week I've OOM'd[4] an old 32 bit jrockit weblogic 10.3 server instance with just 10 concurrent requests to a large richfaces 3 page. 
14. Richfaces 3 is ABANDONWARE [3] -- "old and unsupported" 
15.  Any javascript error while in an oncomplete= is invisible and swallowed.

welp. I forgot the rest of the lessons learned, which is why I'm going to start posting every week. 

[0] http://livedemo.exadel.com/richfaces-demo/

[1] http://blog.jquery.com/2009/01/14/jquery-1-3-released/

[2]https://developer.jboss.org/wiki/RichFacesMigrationGuide33x-4xMigration?_sscc=t

[3]https://developer.jboss.org/thread/241053

[4]OutOfMemoryException. see https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks002.html

[5] https://en.wikipedia.org/wiki/Law_of_Demeter