Google Reader going off-line

Being thrilled was not my reaction when my Google Reader item list informed me that my favorite and only news aggregation is “going out of business”!

GoogleReaderOff-lineI do not understand this decision here. The application is working, the user base is big. So why are they closing it down? Some say that the want to focus on other products, but if that is so then are there not other candidates to close down before we go for this one. Guess not. Some say that they want to focus on Google+, if that is the case then I will be quite disappointed.

So what now? Find an alternative I guess. This will not be easy. But at this point I have “something” that is working.

Its feedly and till now it looks nice. But there are some things that still annoy me. But because I want to do a full post on just feedly going into detail at this point would be counterproductive.

So until next time

Bulk Delete

Maybe it looks like I am having a personal vendetta against OR mappers but this is not the case. In the last week I have seen a otherwise “good” applications performance go overboard because of a misuse of an OR mapper. So I just want to share my findings and maybe save someone, somewhere, some time.

My poison today was the process of deleting lot of records from a simple table. A simple task if you care to get your fingers dirty in SQL, regardless of the flavor.

Let’s say that you have a table of users and you developed a strong hate for the letter ‘H’ (or any other letter in the alphabet). So you woke up with a burning hate of the letter ‘H’ and you can not but delete all users which name starts with the letter ‘H’.

All you need to do is to execute this simple query:

DELETE FROM Users
WHERE UserName LIKE 'H%'

And done…

I agree that this scenario is a little…simple, but even if the structure of the database schema would be a little more complicated you could still do it with very few queries.

Now lets take a look at how this happens with a OR mapper.

With an OR mapper we first have to get a hold of the users we want to delete. The easiest (and most common) way to do this is to load them into memory. That meas that a sql query will be fired to read all users that we are going to delete.

SELECT * FROM Users WHERE UserName LIKE 'H%'

Note: Please take into consideration that this query does not happen if the users are already loaded into memory.

So now you have them in memory. Now you can call Delete on every one of them. Te default behavior of any OR mapper is to fire of individual delete statement, one for each user you want to delete.

DELETE FROM Users WHERE UserId = 1
DELETE FROM Users WHERE UserId = 2
...
DELETE FROM Users WHERE UserId = 5000

There is no arguing that this is less efficient that doing all of it in one big delete statement. The situation gets even worse if there are triggers or a cascade delete :(

But before you start fainting stop think. I found a solution, sadly only for Entity Framework. For the rest I just bypass the OR mapper and use the underlying connection to directly fire SQL statements (how this is done is specific with the OR mapper used).

This breaks the “database agnostic” advantage that OR mapper give you. But to be honest   I think that this advantage is a myth, so I lose nothing here. And besides in all the SQL flavors the vanilla delete statement is the same.

So this is another problem I have with OR mappers and my solution to the same problem. Please keep in mind that this becomes a problem with massive data quantities. What “massive” means for your application is a question that can only be individually answered.

But before I go I want to share my latest story. There was an operation in an application I work on that fired 80k to 100k requests of those 80% to 90% were delete statement. Before I started the operation took 50 minutes and would often fail because of timeouts. After the “optimization” the same operation working with the same data finished in 4 to 6 seconds.

User.Deleted

Interesting title, is it not?

But let me explain. I have noticed, lately, that people have stopped using their brains and have given their data storage over to heuristics. What I mean with that? I mean that more and more people are using their OR mapper to generate their DB schema. And I think that this is not the greatest idea in the world. Maybe this is just me but I hope not.

Point in case is the title.

In the databases “Users” table there may be a column that is labeled Deleted. What this column is doing is telling you that the users is actually deleted. You might be asking yourself that if the user is deleted then why just not delete the user record. Well there are many reasons but the top one is data-integrity. There are ways to work around this but if not pressed by the law we tend to not do it.
For those still in the dark here is a quick recap of why this is done. When the user entry is created we tend to bind other data in the database to that user. When the user then decides to leave our system we tend to keep some of the data he has provided or data that accumulated as a result of him using the system. To keep the database integrity existent we eighter have to reassign all the data to another “bogus” user or just mark the user as nonexistent. We obviously go for the later :)

So the Users.Deleted makes sense in the context of a database but how about in the context of your domain model? In the data domain this makes little sense because if the user is deleted (non-existed for the application) the user should not be loaded into the data domain.

This is just one small example where the data domain and the database schema are not equal. There are countless others and their number will increase with the complexity of the application.

ValyJs Why?

New job, new application,new ideas!

After getting the hang of the application I am taking care of I star noticing the same code repeating. And after taking a look at some other web pages I see the same things repeating.

As a fair warning we are talking about javascript here so the work code is and will be used in its widest definition.

function foo(){
  var element = $("elementId");
  if(element != null){
    //do something
  }
}

or the classic one where an anonymous object is passed to a function and they check for each and every member of the object…

function bar(user){
  if(user.name){
    // use the users name
  }
  // dosomething else
  if(user.surname){
    // now it's safe to use the surname
  }
}

And now imagine that a million times over all the files that compose the client site part of the application.

Yes that makes Dejan quite unhappy :(

So this is where ValyJs comes into play. I would not call it a complete product but I wrote up some helper functions that can aid those common scenarios.

Now you can write it like this.

function foo(){
  valy.executeIfExists({
    elementId : "elementId",
    func : function(element){
      // do your stuff here
    }
  });
}

 

Is this better? I think that it is more readable if nothing else. If nothing else it is consistent.

For the second scenario. We can check for all members up front.

function bar(user){
  valy.exists(user.name);
  valy.exists(user.surname);

  // do your stuff
} 

I to hope that this is better.

There is still a long way to go till I get this to version 1, currently I have it at version 0.1.1 and it is working.

What I need now is some creative input to get something that is actually usable.

So anyway you can go and check it out here: http://code.google.com/p/valjjs/

 

I have to get my stuff together

I fell like I am waking up from a long sleep. After finding no fun in what I do I finally found something that I enjoy doing. After so long I again know why I love the things I do.

Sadly while my head was berried in my depression the world did not stop and a lot of things happened.

The first thing I have to do now is to get my stuff together and continue where I left off.

Day 2 – what the frack happened to you

Well some of you have maybe noticed that there has been no day 2 following my day 1. This is mostly because nothing extraordinary happened on the other days and there were other things that prevented me from writing on my blog.

First of all on Tuesday that week I got a horrific fiver. So that I could barely attend the classes, but I managed to pull it through. So that explains the absence of the missing posts from the 8.19.2011 to 8.13.2011.

When the fiver subsided another thing popped up, or better, popped away. When I had my fiver my body somehow refused to smoke. After I started felling better the non-smoking continued. After smoking for X+ years I just wanted to see how long I can go without.

And to be honest today is day # 20! I do not want to jinx it but I think that there is a slight possibility that I am will join the quitting site of the smoking area.

Until now I have not smoked 400 cigarettes and saved myself around 60 EUR. And I am continually contemplating if I should invest this saved money into Deus Ex for the PS3. To this day I am still thinking about it!

But the back to the reason no posts have been written: The computer was the place where I smoked the most. And so this because the place where I was the least, because the temptation is the strongest here.

I am slowly learning to live without the 5min breaks that were my cigarettes but it is a day to day struggle.

Thx for reading

Day 1 – Pex and Moles

Today was the first day of  my .NET 4.0 training. And I would like to share the highlights of each day. Well at least the highlights for me. So this is the first day of 5.

Introductions and first days are always slow. So this was no exception. There were some basic introduction things for people who have newer came into contact with electricity. But a good 5 hours in something was presented that caught my eye: Pex and Moles.

Pex and Moles are actually two separate peaces of software:

  • Pex automatically generates test suites with high code coverage.
  • Moles allows to replace any .NET method with a delegate.

I am still unsure about the real day-to-day value that moles will offer me. This is not because the software is not up to pair with what I would use but that I am mostly working on code I can change and refactor so that the need for such a tool is not needed. But more on that later on.

First you will need a copy of Pex and Moles. So here is the download link. You can even find a version for the express version (non-commercial). After you have the file just let the installer do its work an tolerate the 2 ~ 3 times your focus will be stolen (it is worth it).

First you need some code to let pex have fun with. I just quickly wrote a little class with one method. And here it is:

namespace PexAndMoles
{
    public class Calculator
    {
        public int Add(int one, int two)
        {
            if(one == 0 || two == 0)
                throw new ArgumentException();

            if(one < two)
                throw new ArgumentException();

            return one + two;
        }
    }
}

There is a reason for all those ifs in there. It is for the sole reason to give pex something to work on :)

So to get started just left-click on the method you want to “work on”. You should see something like this.

Run PEX

Pex will ask you which testing framework it should use. You can choose from all the major testing framework. But to keep it simple I chose to stick with MSUnit.

Select testing framework

After a short time where you are tempted by a “follow us on Facebook” link the results are presented and if you are lucky (depending on the code complexity) pex will find all major test scenarios for your method.

In my case this is what it came up with.

Pex results

And those are all the test scenarios I wanted (or even expected).

Now that you have your tests you want to keep them for later (most probably some sort of regression testing). So pex can help you there to. If you select all created “results” a “Promote…” button will appear. If pressed it adds the “results” as unit tests into your testing project or creates a new one and adds them there.

The code generated is confusing at worst and funny at best. It is not the go-to example of good/clean code. But it is auto-generated and can be regenerated if future changes break the tests. The naming convention is “acceptable”. Before I rant too much here is the code generated:

namespace PexAndMoles
{
    [TestClass]
    [PexClass(typeof(Calculator))]
    [PexAllowedExceptionFromTypeUnderTest(typeof(ArgumentException), AcceptExceptionSubtypes = true)]
    [PexAllowedExceptionFromTypeUnderTest(typeof(InvalidOperationException))]
    public partial class CalculatorTest
    {
        [PexMethod]
        public int Add(
            [PexAssumeUnderTest]Calculator target,
            int one,
            int two
        )
        {
            int result = target.Add(one, two);
            return result;
            // TODO: add assertions to method CalculatorTest.Add(Calculator, Int32, Int32)
        }
        [TestMethod]
        [ExpectedException(typeof(ArgumentException))]
        public void AddThrowsArgumentException547()
        {
            int i;
            Calculator s0 = new Calculator();
            i = this.Add(s0, 0, 0);
        }
        [TestMethod]
        [ExpectedException(typeof(ArgumentException))]
        public void AddThrowsArgumentException81()
        {
            int i;
            Calculator s0 = new Calculator();
            i = this.Add(s0, 1, 0);
        }
        [TestMethod]
        public void Add520()
        {
            int i;
            Calculator s0 = new Calculator();
            i = this.Add(s0, 1, 1);
            Assert.AreEqual<int>(2, i);
            Assert.IsNotNull((object)s0);
        }
        [TestMethod]
        [ExpectedException(typeof(ArgumentException))]
        public void AddThrowsArgumentException470()
        {
            int i;
            Calculator s0 = new Calculator();
            i = this.Add(s0, 2, 3);
        }
    }
}

And that is pex in a nutshell. At least that is what I was able to find out about it in the half day I spend with it.

I know that Moles was not mentioned here but I would like to spend some more time with it before writing about it in more detail, besides the post is long enough.

And that is all the time I have today.

Thx for your time.