March 18th, 2011
You’re probably familiar with it, the build is broken. In a team with a lot of developers committing all the time, this might become a real pain. The build should succeed at least a few times a day otherwise the team might start to feel ‘naked’ not being backed up by a process giving continuous feedback. After all, that’s the whole idea of Continuous Integration platform: commit small working chunks of code, commit a lot, get fast feedback. If the build is broken, fix it asap. New intermediate releases should be deployed on the test environments frequently, so make sure you’re not introducing an impediment.
More on the subject can be found here: http://martinfowler.com/articles/continuousIntegration.html. Interesting read!
Unfortunately, with a lot of developers working on the same project, this might become hard. At least, so it seems.
I’m working at a project with about 30 developers. One can imagine a lot of code is being committed into Subversion every day. To make things more complex, a huge refactoring has been started about 6 weeks ago, moving a lot of code from the ‘old’ architecture into the newly defined bounded contexts. This is done by a ‘side track’ team. Two other teams are still developing new stories. So far, so good. But where’s the hard part you might be wondering?
To me, refactoring is just like writing new code. Make small changes, commit, keep an eye on the CI, make small changes, commit, … . But for some reason, when people start refactoring, committing in small steps seems to be very hard. As a result, big commits are submitted into SVN, other people get huge merge conflicts. The build starts failing, tests are broken, the application isn’t even booting anymore.
So, the 3 teams on the project decided to introduce the build token. As long as the build token is taken by the build master (indeed, we introduced a build master to keep an eye on the build.. *sigh*), no one is allowed to check in. As that didn’t work very well – the build is till broken -, the teams went a step further. A build queue was introduced and only the one with the build token is allowed to commit. But he should run his unit and integration tests first, taking about 25 minutes.
So, when receiving the commit token, one should update his code, run the build with all the tests and when it succeeds, he is allowed to commit.
IMHO, all that is kinda stupid. Creating a queue results in even larger checkins. People are getting blocked. People are syncing all the time to avoid merge conflicts and have to run their builds all the time on their local machine. So their watching build traces all the time. Off course, the build is succeeding on the CI now, but at what expense ?!? I mean, why do we even bother having a CI? We already know the build still succeeds as it builds on everyones machine before committing. Why bother maintaining and watching it ?
As a side note, for those who ever had to work with Visual SourceSafe (I felt your pain), one had to lock a file before something could be changed. Really annoying when working with a lot of people in a team. But this situation with the commit queue is much worse! Not a few files are locked, but the whole project is locked!
Again, IMHO, my only advice to brake this stupid spiral is to return to committing small chunks of code. Not running all the test locally (only the ones involved). Not caring about a stupid commit token, BUT keeping an eye on the CI. Every developer is responsible for his acts. He should feel bad about breaking the build. And this only works with fast feedback from the CI in order to avoid one forgets he committed some stuff and should keep an eye on the CI. Staged builds are the answer here. Or if possible pre-commit builds on the CI. TeamCity already has this feature.
So in short, let the CI work for you, don’t work for the CI.
Commit small. Commit fast. Be responsible. Feel guilty. Fix asap.
Tags: CI, craftsmanship, software processes Posted
in java | No Comments »
February 21st, 2011
If you’re following twitter or reading some blog RSS feeds, you’ve probably noticed the Agile Manifesto had a little birthday party, it became 10.
The question is however, is there still a reason to celebrate? ….Yes and No.
Yes, because the Agile Movement that was started by a few anarchists made it to the de facto standard in software development country. A major part in our business says they are doing it, are adopting it or are considering the big jump towards it.
No, because everyone is doing it, is adopting it or is considering to make the big jump. Uhm, isn’t that a paradox? Nope. Reading the same RSS feeds and tweets, one can feel the pain of the anarchists of ten years ago when they look at what happened with their Agile Movement. Reading this article of ten years ago again, you might understand why. As it is well explained in this blog by William Pietri a lot of the agilists out there have missed the point completely. The industry picked up the Agile idea and before knowing well what it is all about, the Agile word became the next buzz word on the block, resulting in being a victim of its own popularity.
Then the Craftsmanship movement came along to act as a counterweight against the failing agilists. Again, the ideas of the movement are rock solid and to the point, finger pointing the spots where the want-to-be-agilists fail. But again, the danger to be adopted by the mass industry lies behind every corner. How could one be against doing the best you can day after day? How could one advocate not to test as much as possible? How could one say not trying to involve his client as much in the project to achieve the best productivity and product value as possible?
My point is, once marketing and the money will discover it, it will be doomed as well. Unless, unless the craftsmanship movement can keep its anarchistic nature. Unless it can stay underground and irritate the would-be-agilists, reminding them what is all about day after day. Asking them constantly why we have a failing build for 25 days in a row. Why our scrum board is full of clutter and why it takes all this time to manage it? Why the documentation is so scarce? Why we are pairing day after day, even for the simplest tasks (because XP says you have to ?) Why the stand up meetings take more than a few minutes? Why the code quality is so poor and nobody is seeing it? Why you have no metrics telling you the quality of your code? Why retrospectives feel like meetings you have to attend because you have to, but whatever the resulting actions are, nothing is going to change? Why is one developing on the worst possible machines out there and that are even more ruined by security rules and virus scanners so one is looking at the damn hourglass pointer for the major part of his days? Why is the client not more involved? Oh, he is.. right, a lot of people are involved analyzing the stuff we have to develop. But how about returning their stuff back to sender when it’s crap and getting the developers involved as soon as possible. And please skip the shitty unprepared meetings resulting in more questions than before! And what about the communication gap? Why does the team not feel like a team?? Why do we need a complete sidetrack team to reorganize our code in a different architecture – will it improve quality / development time (interesting article)?
Is all that being Agile ?
These are only a few questions that come to my mind when I think of some projects I worked/I’m working on. But being a true professional, maybe I shouldn’t be writing this. It’s easy to find out what projects/firms I’m talking about. But maybe, being that same professional, I should be writing this. Maybe it starts the discussion again. I don’t mind being wrong and having to admit so, as long it keeps everybody sharp.
I just can’t live with the idea working for a company that claims to be one of those ultra agilists around in my country, while they simply are not. To be fair, they come closer than the most out there, but unfortunately, no cigar.
And to finish this post, an interesting thought: my first project, which had waterfall project management written all over the place, just might have been more agile than the project I’m doing right now.
Tags: agile, craftsmanship, manifesto, professionalism, software processes Posted
in java | No Comments »
October 1st, 2010
Today I had to fix a bug in a web application deployed on Tomcat saying that a user had to wait too long before he received the message on his screen ‘The mail has been sent’.
Wild ideas like queues, saving the mail to the database and process it with a separate polling service or posting it to a separate http request were the first solutions I could think of. But that would have been some overkill as the mail sending functionality isn’t used that much.
So I started googling and found out Spring 3.0 has some nice task management support. You can read all about it here Spring 3.0 Reference documentation
I was already using the MailSender interface of Spring, so I implemented a different implementation that sends the mail in a different thread that it gets from a thread pool. There is even a waiting queue.
And this is how it looks like:
package com.idevelop.altiplano.services.tasks;
import javax.annotation.Resource;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.core.task.TaskExecutor;
import org.springframework.mail.MailException;
import org.springframework.mail.MailSender;
import org.springframework.mail.SimpleMailMessage;
import org.springframework.stereotype.Service;
/**
* Task executor for sending mails asynchronously.
*
* @author Steven Willems
*/
@Service(value = "asyncMailSender")
public class AsyncMailSender implements MailSender {
@Resource(name = "mailSender")
private MailSender mailSender;
private TaskExecutor taskExecutor;
@Autowired
public AsyncMailSender(TaskExecutor taskExecutor) {
this.taskExecutor = taskExecutor;
}
public void send(SimpleMailMessage simpleMessage) throws MailException {
taskExecutor.execute(new AsyncMailTask(simpleMessage));
}
public void send(SimpleMailMessage[] simpleMessages) throws MailException {
for (SimpleMailMessage message : simpleMessages) {
send(message);
}
}
private class AsyncMailTask implements Runnable {
private SimpleMailMessage message;
private AsyncMailTask(SimpleMailMessage message) {
this.message = message;
}
public void run() {
mailSender.send(message);
}
}
}
Unit test and corresponding application context:
package com.idevelop.altiplano.services.tasks;
import javax.annotation.Resource;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.mail.MailSender;
import org.springframework.mail.SimpleMailMessage;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
/**
* Test class for {@link com.idevelop.altiplano.services.tasks.AsyncMailSender}.
*
* @author Steven Willems
*/
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration
public class TestAsyncMailSender {
@Resource(name = "asyncMailSender")
private MailSender mailSender;
@Test
public void testSend() throws Exception {
SimpleMailMessage[] mailMessages = new SimpleMailMessage[50];
for (int i = 0; i < mailMessages.length; i++) {
SimpleMailMessage message = new SimpleMailMessage();
message.setSubject(String.valueOf(i));
mailMessages[i] = message;
}
mailSender.send(mailMessages);
}
}
In our application context, we define ThreadPoolTaskExecutor who will manage the thread and the waiting queue.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:context="http://www.springframework.org/schema/context" xmlns:p="http://www.springframework.org/schema/p"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">
<context:component-scan base-package="com.idevelop.altiplano.services.tasks"/>
<!--
Thread pool definition for tasks to be executed asynchronously.
-->
<bean id="taskExecutor" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" p:corePoolSize="5"
p:maxPoolSize="10" p:queueCapacity="100" p:waitForTasksToCompleteOnShutdown="true"/>
<!--
Mail sender
-->
<bean id="mailSender" class="com.idevelop.altiplano.util.DummyMailSender"/>
</beans>
The DummyMailSender doesn’t really send a mail, it just prints out the subject of the mail message.
package com.idevelop.altiplano.util;
import org.springframework.mail.MailException;
import org.springframework.mail.MailSender;
import org.springframework.mail.SimpleMailMessage;
/**
* Dummy mail sender
*
* @author Steven Willems
*/
public class DummyMailSender implements MailSender {
public void send(SimpleMailMessage message) throws MailException {
System.out.println("sending mail message '" + message.getSubject() + "'");
}
public void send(SimpleMailMessage[] simpleMessages) throws MailException {
}
}
And to prove it did it asynchronously, here is the output:
sending mail message '0'
sending mail message '1'
sending mail message '2'
sending mail message '3'
sending mail message '4'
sending mail message '5'
sending mail message '9'
sending mail message '8'
sending mail message '7'
sending mail message '6'
sending mail message '10'
sending mail message '11'
sending mail message '14'
sending mail message '15'
sending mail message '13'
sending mail message '19'
sending mail message '20'
sending mail message '21'
sending mail message '22'
sending mail message '23'
sending mail message '24'
sending mail message '25'
sending mail message '26'
sending mail message '27'
sending mail message '28'
sending mail message '29'
sending mail message '30'
sending mail message '31'
sending mail message '32'
sending mail message '33'
sending mail message '34'
sending mail message '35'
sending mail message '37'
sending mail message '38'
sending mail message '39'
sending mail message '40'
sending mail message '41'
sending mail message '42'
sending mail message '43'
sending mail message '44'
sending mail message '12'
sending mail message '36'
sending mail message '48'
sending mail message '49'
sending mail message '18'
sending mail message '17'
sending mail message '16'
sending mail message '47'
sending mail message '45'
sending mail message '46'
Again, very simple and it just works. Thank you Spring
Posted
in java | No Comments »
July 30th, 2010
Recently I re-experienced some classic but annoying problem we web developers all have run into at least once. And like for the most of those kind of problems, we do a quick google search, find the answer, implement it and try to remember it. Sometimes we succeed doing so, sometimes some other strange problem overwrites that spot on our hard disk or like for the most of these problems and their solutions, time starts switching bits randomly in our head so we first start fail at reconstructing the total story and eventually start forgetting the whole story..
That being said, the problem I ran into was disabling the browser cache for dynamic generated html pages. That’s easy I hear you thinking and your right, it is. Only, how do you achieve it with SpringMVC 3.0?
I remembered vaguely something about cache parameters in the header of the response. So I first tried adding the following parameters in the section of my html page.
<meta http-equiv="Expires" content="0"/>
<meta http-equiv="Cache-Control" content="no-cache"/>
<meta http-equiv="Pragma" content="no-cache"/>
A good attempt, but all of my browsers (Safari, Chrome, Firefox – didn’t try other ones) kindly ignored me. So I googled and I learned that most of the browsers ignore these settings and even if some browser would take them into account, proxy servers never will. Proxy servers don’t read the HTML itself, they look at the HTTP headers and see the HTML as some response stream they don’t look into.
Aha! Let’s see what kind of HTTP header parameters we can set. Again my friend Google learned me:
Pragma: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: max-age=0, no-cache, no-store
So now the only problem remained how to get these parameters on the response stream of my SpringMVC 3.0 Controller objects. If you are using Spring MVC with xml bean declaration, your controllers will inherit from the org.springframework.web.servlet.support.WebContentGenerator class where you set some properties to enable the above mentioned parameters. But when you are using annotations like I do, you’ll have to figure out something else..
You could add the HttpServletResponse as a parameter to your @RequestMapping annotated method and manually add the parameters for each request. It will work. But I don’t like it. One of the ideas behind annotating the controller objects was to make unit testing easy by excluding all the HttpServletRequest/Response stuff so one doesn’t have to mock them. So I don’t want to pull them back in.
I fixed my problem by using the org.springframework.web.servlet.mvc.WebContentInterceptor. The interceptor extends the WebContentGenerator class so it has the setter methods again I need to enable the header parameters. In my spring config file I have now this:
<context:component-scan base-package="com.idevelop.abc"/>
<mvc:annotation-driven/>
<mvc:interceptors>
<bean id="webContentInterceptor" class="org.springframework.web.servlet.mvc.WebContentInterceptor">
<property name="cacheSeconds" value="0"/>
<property name="useExpiresHeader" value="true"/>
<property name="useCacheControlHeader" value="true"/>
<property name="useCacheControlNoStore" value="true"/>
</bean>
</mvc:interceptors>
Again, it works, but I don’t like it.
Now I have enabled on all the requests that are handled by SpringMVC the ‘no cache’ parameters. Not something you would like to do in a real application. E.g. some pages can be cached, some can live for e.g. 1 hour, some for a week, and so on.
To resolve this, you could extend the WebContentInterceptor and write your own interceptor and tweak your cache management based on the incoming urls. But again, I don’t like it.
What I would really like is a new annotation: @Response And I would put it on my @RequestMapping methods. After all, if your handling the request with annotations, why not the response ?
The parameters it should take could be the same as the ones in the WebContentInterceptor to start with.
Such an annotation could look like this:
@Target({ElementType.METHOD, ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Mapping
public @interface Response {
boolean useExpiresHeader = false;
boolean useCacheControlHeader = true;
boolean useCacheControlNoStore = true;
int cacheSeconds = -1;
}
What do you think? Good enough to put it as a feature request on the SpringMVC jira ?
Tags: Spring MVC Posted
in java | 10 Comments »
February 20th, 2010
.. if the engineering part is neglected.
Wherever I come, I’ve been advocating the need of Continuous Integration (CI), Unit Tests (or even better TDD), Code Reviews, … Unfortunately, the people who needed to hear it the most, where ignoring me. I was the new guy, the one who has adopted the ‘modern’ way of working. The one that brings more work and new stuff and things we don’t know. Luckily, I also have booked some successes introducing CI, convincing the people to work against interfaces, to use dependency injection (DI) and stuff like that. But as long the whole team doesn’t go for it, it is as weak as the weakest link.
So what’s the link with software processes like The Waterfall model, or Scrum or CMMI or RUP or Crystal, or..? Well, first of all, a side note, please do not use the waterfall model! Even the first paper that describes the waterfall model, also describes it as “risky and invites failure”. It doesn’t embrace change. And in software development, change is around every corner.
But back to the topic of this post: why do all processes fail? Because the software engineering is not in place to support it. Adopting a process/methodology has in most of the cases a positive effect. It’s new, people have the feeling it works, they are motivated again.. But soon, they are confronted with a higher load of bug fixes, or change requests arrive and they are hard to implement or a team member is replaced by a new one and introduces too much bugs because of the lack of unit tests.
So my point is, good software engineering is really really needed. It is not a nice too have. It’s not needed just to be able to say: our code is fully documented, or has a 95% test coverage (BTW, the % itself doesn’t matter, 80% is off course better than 5%, but if you have 80% of tests testing getters and setters, I would have preferred having the other 20% testing the real stuff). So these things need to be in place for a reason. They support the software process.
When a new team member arrives, he doesn’t know anything about the code. When he starts changing things, how can he now he brakes stuff? -> Unit Tests. How can he know that right away? -> CI. When a new feature needs to be implemented, e.g. a new CSV file importer needs to be written -> Use interface. / Use DI. When a new version of a third party library comes out, how do I know it doesn’t brake stuff? -> wrap it and test the wrapper. How can we get the team aligned? Have them write consistent/qualitative code? -> Code reviews. And so on.. Doing all this, the chance of introducing bugs, tightly coupled code, or unmaintainable code is reduced to a minimum (you’ll never ban it completely).
This post was triggered by a post of Ayende: http://ayende.com/Blog/archive/2010/02/20/nice-process-but-what-about-the-engineering-bits.aspx. He says what I have been thinking for a while. Always nice too read your own thoughts
Tags: professionalism, software engineering, software processes Posted
in java | No Comments »
|