A mathematician at risk

This is how advertising should work

2010-02-06T18:49:00.000-08:00

My new landlord sent me an email today to let me know the dimensions of my windows in the apartment I'm moving into. But what do I know about drapes, or blinds, or whatever one uses to cover windows.

But what's that at the top of my webmail inbox?

Now I could treat this as an invasion of privacy or being over exposed to advertisements. But honestly, this is an ad that makes my life easier.

I haven't looked at their product/prices yet so in no way should you take this as an endorsement. However, point one in their favour is that they are paying for me to receive information. Clearly, my receiving this info is mutually beneficial yet they are incurring all of the costs. That's the right attitude as far as I'm concerned.

Eating Crow

2010-01-27T14:28:00.001-08:00

If I'm going to write a prediction post it's only fair that I write a follow up to rate myself.

The product will be too expensive. Obviously I was very wrong here. There were a lot of leaks that the price point would be a lot lower than people expected and I clearly ignored them. But let's not pretend the unit is $500 either. 16GBs is clearly not enough storage to use the iPad in any reasonable way. And personally I'm resistant to paying for another monthly data bill. But we aren't talking about apple hifi or iPhone gen 1 prices here.
Most developers will be locked out, except perhaps in a ridiculously restrictive way. Honestly, my expectation was that the only way you would be able to get applications onto the device was through the app store. But I over stated because I didn't expect iPhone apps to be compatible (and I assumed the API would be released at WWDC). I figured even if there were app store applications available immediately it would seem pretty pathetic because I was expecting a device that looked more like a mac than an over-sized iPod. All that said, I'm still going to give myself half points for this one, rather than no points. Just because we've gotten used to the idea that you have to get permission to put an application on your own device doesn't mean it's not ridiculously overly restrictive.
Content channels will be incredibly locked down. Again, I was expecting a more general/mac like device instead of an over-sized iPod. If this had been a mac tablet then it would have seemed absurd that it couldn't play divx. As a large iPod we've been conditioned to expect it not to play divx. But I think I really have to give myself a zero on this prediction simply because apple went with a standard and used epub. However, here are some things to consider. My guess is apple will treat epub like aac: you will be able to install non-drm'd epubs from anyone but only apple will be allowed to offer drm'd epubs. Where does this put amazon? I'm sure the kindle app will still be available but there will be advantages to using the native iPad book reader. Amazon's content deals won't let them sell drm free books and apple won't let amazon sell drm'd books into the iPad book reading app. So the only major seller of content of books for the iPad will still be apple.
There will be some incredibly obvious feature that is inexplicably missing. Really this is a silly prediction. Of course features are inexplicably missing. Because the explanation is available to those who are working in the real world who are dealing with timelines and pricing restrictions. So yeah, there is no gps in the non-3g version, no camera, no multi-tasking, and no flash. And there are probably many more. So I get full points but how could I not.

So why were my predictions so wrong? Am I just an idiot? I think all my predictions fairly cited apple's history for first gen products. Perhaps what I missed is that maybe this isn't a first gen product. This really is just a big iPod touch. Book store is new and so are these iWork apps but really they are just riffs on current businesses. And we haven't seen how they will pan out yet.

Careful overextending that model

2010-01-25T18:43:00.000-08:00

In my current work I spend a lot of time working with models that try to predict the likelihood of a customer's future action. From time to time we have a need to model something new quickly and it is very tempting in those cases to take a different but similar model and see if it can simply be re-aligned to fit the new circumstance. Sometimes this works fine but it is interesting how often this fails dramatically.

But I have to be careful about writing about specific examples from work. So let's consider a hypothetical example instead to see the danger of assuming a model extends.

How about consider a software company who has started a pilot program of giving potential clients six week trials of their product. Now they would like to know as soon as possible what sort of customer is most likely to convert so they can focus their sales force most effectively. Rather than wait six weeks for results they decided to try to observe who will convert in the first week (ie a target of one week conversion). That way after one week, plus development time, they will have a model that they can start using. While this model won't have much accuracy in predicting the probability of converting in six, surely it will rank order correctly with a model that used a full observation period. After all, if a customer that doesn't convert in week one has similar characteristics as someone who does then it seems fair to conclude they just need a little bit more of a push to become more like their cohorts.

However, this model fails disastrously. Those who were supposed to be the most likely to convert ended up being the least likely. So the model is revisited to see what could have gone wrong. It is discovered that the biggest drivers of the model were

lots of use of the software
lots of interaction with the sales department (asking questions about price, etc)
indicates on questionnaire that the software is to be used on an urgent project
asks a lot of questions about the features that are locked by the trial.

And that's when the analyst hypothesizes his mistake. Instead of targeting for the desire to convert, he suspects that he's targeted for the need to make a decision quickly. So those who were rated highly by the model and didn't convert in the first week, didn't convert because they had decided in that week that they would never convert.

To test this theory the analyst checks whether those who did not convert, but were predicted to, had bought the competitor's software (the trial included some spyware so he can check for such things). And there is the answer. Not only had they bought the competitor's software in large numbers, but usually with in the first week of the trial. Those that were considered the best leads had, unbeknownst to the company, had already ruled themselves out of being future customers.

Now real life examples are rarely this clean cut. In reality, the population with a high likelihood of converting in the first week would probably be made of up a mixture of types. Instead of moving as a group to being the worst converters, you would probably some sort of indeterminate result.

So how could this mistake have been prevented. Well first by validating that there was some justification that the model could be extended. Perhaps he could have validated against the second week to find out sooner that he had made a mistake. But more importantly, the analyst waited too long to try to understand his variables. When making a model like this it is important to try to understand why each of the drivers of the model are actually predictive. And then test the relationships that you believe exist. That is to say construct a narrative that explains why your variable predicts the way it does, consider the other consequences of this narrative, and see if these other consequences exist in your data.

Even though we aren't wearing lab coats it is a good idea to keep in mind that we are still doing science. The scientific method makes a pretty good guide.

Implement US health care reform at a state level

2010-01-24T19:03:00.001-08:00

Slate asks how the Democrats can still implement healthcare reform. This was my response:

Consider the major points of the senate bill: state level exchanges, individual mandate, no refusal based on pre-existing conditions, and subsidies paid for by Cadillac insurance tax. Assuming the tax and subsidy is distributed equally, the benefits for any one state implementing this bill is not affected by how many other states implement this bill.

And since health insurance companies are not allowed, and would not be allowed, to operate in one state and insure someone in another, there is no issue of a loss in cost savings due to smaller pools being insured.

So if there is no advantage to implementing this nationwide, instead of in just the states that can pass it, why not just pass the bill at a state level in states where it can pass. A domestic coalition of the willing.

There is a non-zero sum game in game theory called stag hunt which has two defining characteristics: a single defector is able to lower the payoff of the co-operators, and the single defector lowers his own payoff by defecting (unless there is another defector). It is this situation that justifies modern democracy where the majority is able to enforce the co-operation of defectors (whose defection wouldn’t even be in the defector’s own self interest).

But this isn't stag hunt. A single defector, or many defectors, has almost no affect on those that co-operate. So implementing at a state level may even be more democratic in this case. After all, shouldn’t individual choice be preserved where possible.

There may be two migration issues to worry about in this strategy: those with pre-existing conditions moving into states that implement this bill, and healthcare providers moving out. However, there are lots of other ways that entitlements vary from state to state, so if migration isn’t already a strategy for those seeking extra entitlements there’s no reason to think this entitlement would be any different. And we’ve been told that this bill won’t hurt healthcare providers so there should be no reason for them to leave.

Finally, consider our healthcare in Canada. While we have broad laws at a federal level requiring single payer and transferable health insurance, healthcare itself is actually implemented at the provincial level. This is because (a) the provinces were considered too varied to manage all of healthcare from Ottawa and (b) it was thought that trying to manage the healthcare of 30 million people in a single institution is a task that only a crazy person would attempt. 300 million is a bit more.

My pessimistic/realistic apple tablet predictions

2010-01-20T19:38:00.000-08:00

So I wanted my next blog post to be a little less controversial but I think I'm about to kick the biggest hornets' nest yet: apple fan-boys. But the next apple announcement is around the corner and I don't have much longer to be a wet blanket.

Note, that I'm basing these predictions completely on apple's track record with first gen devices. This isn't what I want to see happen, just the pattern I'm used to from apple. Also be warned; I'm a bit rant-y.

The product will be too expensive. What did the iPod, iPhone, apple speaker and appleTV all have in common on day one? They looked fun but they were way too expensive to consider actually buying under normal circumstances. In each case multiple hundreds of dollars to expensive. Now, they come down in price eventually (assuming the product isn't discontinued first). And they can get away with the high price because there is a cluster of consumers at the high end of the demand curve that are sufficient to buy all of their initial supply, which is usually pretty small. But I don't think it makes sense to expect a sane price.
Most developers will be locked out, except perhaps in a ridiculously restrictive way. I've heard various predictions on how developers should be excited because they will have a whole new way to make a ton of money. But apple has never let developers make money on their products on day one. And when you are allowed it has to be exactly on their terms. When the iPhone came out it was just assumed that anyone who was buying one would jailbreak because apple had locked it down so much. This is still true with the appleTV. This is the company that made an iPhone that had a recessed headphone jack so the vast majority of third party headphones wouldn't work with it. And the company that disallowed the iPhone podcaster app, allowed it, then re-disallowed it, put it in a three month penalty box, then re-allowed it again.*
Content channels will be incredibly locked down. In fact all content will go through apple. This one seems obvious to me. The appleTV is really the only set top box left that can't stream netflix in the US. Neither the iPhone nor the appleTV have an approved way to play divx files. Their may be a crack in this lockout similar to podcasts on the iPod. But if you want to charge for your content you are going to need to go through apple to get onto their device. And then you'll have to wait (and as I understand it, wait and wait and wait) to get paid by apple. So I expect predictions that this will be the perfect universal content device to be very very wrong.
There will be some incredibly obvious feature that is inexplicably missing. The appleTV doesn't have a tv tuner, and no approved way to add one. The iPhone has had bluetooth from day one but no way to use an external bluetooth keyboard. The iPhone took multiple generations to get cut and paste. The apple mouse still doesn't have a simple second button. (No, that doesn't count. I said a "simple" second button.)

So might these predictions be wrong? Sure, and I hope they are. But if they are wrong it's because apple has decided to alter their behaviour. There are those who expect this to be the perfect universal device that can be crafted into whatever they need a tablet for. I have no idea why they think such a device could ever be a first gen apple product.

*Ok, I need to rant about this a little more. Let me be clear: apple hates developers. I used to think they were generous because they gave away xcode. But then I realized that they aren't giving away xcode, they are exclusively bundling the only IDE that can develop for their platforms with their computers. So as a developer, you are allowed to develop for the iPhone, but to do so you need to buy one of their computers, only develop on that computer, and then seek approval for what you've done to get it into the store. There, I'm done ranting.

When criticizing Israel is anti-semitism

2009-11-21T00:29:00.000-08:00

A friend of mine linked to a story in the tyee that I responded to on facebook. I've decided to copy that response here.

While it is certainly true that not all criticisms of Israel are acts of anti-semitism, there are those who call themselves critics of Israel who really are spouting hate and anti-semitism. To understand this you have to appreciate just how our people, our land, our history, our customs and our spirituality are interwoven in Judaism. Somewhere just under half of all Jews are Israelis; and a high majority of Jews in the Diaspora self identify as supporters of Israel.

Absolutely you can find examples of Jews who reject Israel - even groups of them. You can also find Jews for Jesus. I don't think anyone would say that because of this the Jewish people are ambivalent on whether Jesus has a role in Judaism. It is simply noted that there are exceptions.

Of course it is reasonable to question and criticize actions of the government and the state. Ex-prime ministers of Israel do this. The current Israeli president has done this in the past. And of course Israelies don't vote as a monolithic block. In fact in a first past the post like system Netyahu would probably not be Prime Minister right now; and Lieberman would certainly not be a serious player.* (This is not to say I think that this government is illegitimate in anyway.)

What crosses the line into anti-semitism is when people say (and I've met these people in university) "I don't have a problem with Jews; it's just Israelis I hate" (and they felt comfortable saying this to my face). And it is easy to find examples of people crying "Zionist thieves" at rallies. To be clear, when you express hate for Israel you are expressing hate for Jews. When you call Zionists thieves you are calling me and my family criminals.

So yes, not all criticisms of Israel are anti-semitism. But why let the anti-semites hide amongst you? Why be an apologist?

*This is one of my personal complaints of the Israeli government; in attempt to be more in line with academic theories of "democracy" Israel has a proportional representation system. Because of this voters aren't required to seek out a compromise with their vote in order to influence the government. This has resulted in Knessets made up of bizarre and unstable coalitions with those on the edge getting to hold the balance of power. First past the post would solve this. STV would not.

New job/blog title

2009-10-30T14:43:00.001-07:00

I recently changed my blog title, and most people I know already know why. It's because I recently changed my career which I think affects my perspective, and probably the topics I will blog about.

I'm now a credit risk analyst at a Canadian bank. Which means I look at the behaviour of our credit card customers and work on creating scores that try to predict their future behaviour, and the future behaviour of applicants. I won't mention the name of the bank because I don't want to speak for them and I certainly don't want to pop up in searches for them.

In the process I'm building up my modelling chops, learning about companies like fico and sas, and getting a new look at how various people use their credit. First big lesson, various people vary a lot. There's no such thing as a perfect or invincible score because people of all types and behaviours go bankrupt all of the sudden. For some behaviours that might be a 1 in 100,000. Which sounds like an absurdly low risk, but for the individuals that realize that risk it is a 100% reality.

Anyways, I feel like I'm back with my mathematical roots, though I'm taking more of a statistics bent this time.

How getting vaccinated protects others

2009-10-28T16:14:00.000-07:00

In my last post I explored a possible model for the flu and how different levels of immunity had social impact. Here I want to drill down more on how getting vaccinated not only protects yourself, but even those who aren't immune.

Here I have a graph of the expected number of people to be infected for various immunization levels.

Here we see if no-one is immunized we have an expected infection level of 61 but if 30 people are immunized that number drops to 17.

Perhaps this isn't very illuminating. I mean of course the more people who are immunized the smaller an outbreak will be, since those who are immune won't get infected. But even more than that is happening. Let's look at the chances of a non-immune person getting infected given various immunization levels.

So here we see that an individual has a 61% chance of infection given no immunity in society. But as her neighbours get immunized her chances of getting infected drop, down to 9% when 50 other people are immunized. This is because each person who is immune (a) reduces the chances of an outbreak and (b) increases the chances that the outbreak will burn out sooner.

There's an even more powerful way to see this by looking at the multiplier effect of getting vaccinated. The multiplier is how much the expected number of people infected drops divided by the amount that the personal risk drops. For example wearing a seat belt has a constant multiplier effect of 1. If wearing a seat belt reduces my chance of harm from 1% to 0.5% then we expect the expected amount of people harmed to drop by 0.005. You wearing a seat belt protects no-one but yourself.

However, not drinking and driving has quite a high multiplier since the accident caused by a drunk driver is quite likely to harm more than just the drinker.

Here's the graph for the multiplier effect of getting vaccinated in the model.

For most of the graph the multiplier effect is somewhere between 2 and 4. So by this model, you getting vaccinated passes on the protection in at least equal levels. By making yourself safe you are protecting others in the process.

Lowering the transmission level also has a similar effect to raising immunization. Stuff like washing your hands helps in this regard (see the post from the spring).

Why is h1n1 a big deal? (and why you should get the vaccine if you can)

2009-10-25T17:19:00.000-07:00

First off, I need to get this out of the way

Bill Maher is an idiot.

I think it's appropriate for anyone writing about H1N1 and vaccines right now to also begin their post with that comment.

With that out of the way, it seems there is a lot of confusion out there about what make H1N1 special this year. Depending on where you read it seems that it is either more infectious, just as infectious, or less infectious than the regular flu. Same goes for the intensity of the infection. Adults in their 20s and 30s may be most susceptible, or it may be the regular group of immune deficient, elderly, children and pregnant who have the most to worry.

I can't answer any of these questions for you. But I can show you why you should care about this influenza virus, and if at all possible why you should get the vaccine (if you are in Canada you definitely can get the vaccine; there was enough preparation to make sure that there is enough vaccine for everyone).

The key difference about H1N1 is that virtually nobody is immune to it. With the regular seasonal flu mutation is small enough that year over year there is some immunity that carries over, and this is enough to get a herd immunity effect. I think that this is one of those points that seems so obvious to epidemiologists and infectious disease specialists that they forget to mention it most of the time. But the one time I did hear it mentioned I was reminded of a very simple epidemic model I had constructed a few years ago.

The model attempts to mimic the behavior of a 100 person clique (100 people who are all equally connected to each other) after an infected patient 0 is introduced to the group. I assume that for every day a person is infected they have a 0.3% chance of infecting any other person in the clique; that is there is a 3/10ths of a percent chance that the infected individual will interact with someone else and that interaction results in a transmission of the infection. And I assume that an individual is infectious for 3 days.

Disclaimer: There is no reason to believe any of these assumptions estimate anything about reality. In fact most social networks aren't made up of cliques of this size - you actually have some people who are more connected and some who are less. And there's no reason to believe that your butcher goes to the same banker that you go to (which is necessary for a clique). Plus the 0.3% and 3 days numbers are completely made up numbers. This model has no predictive power.

However, I'm not trying to make a prediction of the intensity of an epidemic; I'm simply want to show some of the characteristics or rules of an epidemic. To make a somewhat opaque metaphor, I wish to show the shape, not the size, of an epidemic. Specifically I want to show that (a) herd immunity is real and (b) once an outbreak begins it is hard to stop.

To see this let's look at the cumulative distribution given for various numbers of vaccinations. These show the probability of how many people will get infected before the infection burns out (that is until no-one is left infected). These values were generated with fortran 95 code using 500,000 trials per vaccination level (pro tip: learning fortran is a stupid way to spend an afternoon).

What this cumulative distribution shows is the probability that an outbreak can be bounded by a particular size. So the chances that less than 45 people will be infected when 20 have been vaccinated is about 60%.

Notice that in the cumulative distribution, the more people vaccinated (or those who come in immune) the higher the graph starts on the left. This shows that the more people you vaccinate the higher the chance that patient 0 won't be able to infect anybody else. This is kind of obvious when you think about it. The more immune people out there the more likely the people patient 0 interacts with are immune. So if nobody is immune there is only a 14% chance of no further infection. But if you immunize 20 people that number jumps to 21% (again, these numbers aren't at all predictive as to the size of the effect).

Also notice the more people you immunize the quicker you get to 1 in the cumulative graph. This is herd immunity fully kicking in. So if no-one is immunized herd immunity will almost certainly mean no more than 90 people will get infected. But if you immunize 40 people it knocks down the maximum size of the epidemic to about 40 people.

Also note the flat line in the middle of the 0 vaccinated cumulative plot. What this shows is that once 8 or 9 people get infected it is very unlikely for the infection to burn out until it reaches at least 60 people. The more people who are vaccinated this line is shorter and more steep. This means that the more people who get vaccinated, the less capable an outbreak is of sustaining itself.

All this to say that introducing a new flu, where very few people are immune, has a big effect on the number of people infected. The less people who are immune the more likely an outbreak is to happen and the larger that outbreak is expected to be.

In the next post I will examine how much getting vaccinated assists those who don't get vaccinated.

Benchmarking safari 4 javascript with dropsecret

2009-06-10T20:47:00.000-07:00

As a follow up to my earlier benchmark post I'm adding numbers for Safari 4 on a mac. Note that this is a completely different box than the previous benchmarks so really shouldn't be compared, however firefox 3.0 is also here to give some ability to compare.

Firefox 3.0: 10.802 seconds
Safari 4: 14.025 seconds

Of course this is still only one test, but it's arguably a real world test.

Becoming a mathematical generalist

2009-05-29T13:00:00.000-07:00

I've been thinking a bit about an old blog post on less wrong by Eliezer Yudkowsky and trying to determine whether I disagree or agree with it.

The post calls out for a book to be published that will shallowly cover the basic mathematics of as many scientific fields as possible.

My initial problem with this idea is that math is really broad. Really really broad. So much so that it seems to me that the proposition that "The Simple Math of Everything, written for people who are good at math, might not be all that weighty a volume" is a bit of an absurd statement.

And even if we are to just consider the math used in sciences it seems rather silly to believe that one could read one book and then become a broad generalist. If I were to imagine someone who is like Yudkowsky's aspirational generalist I can't see such a person saying "last February I decided to study this text and now I know the basics of everything." It seems that a generalist, no matter how shallow, is a product of years of curiosity.

However, I am extremely sympathetic to the goal of generalism. I would just propose a different system to get there. I would begin with finding a local group of 6 or 7 who are also interested in this goal and schedule weekly two hour sessions with this group.

In the second hour of these sessions one member of the group will pick out a field or domain which she will present. Now there is only an hour so a little discipline will be necessary. I wouldn't spend anytime trying to explain why the audience should be interested in this topic; this is a group who desire to be generalists so they are expected to be interested in any topic new to them. And you can probably ignore history and biographical information. And please immediately kick out anyone who has any sentence resembling the form "the dictionary defines <> as <>."

I tend to like a problem based approach to learning instead of a solutions based approach. So I would begin with the solvable problems of the domain and then onto the basic methods and formulas used to solve them. And finally a little bit of homework using the methods.

Then the first hour of the following week can be used as discussion of the previous week (or presenting solutions to the homework if discussion requires a little push to get going).

How does the presenter learn the basics of her domain in the first place? In conventional ways I guess: find an expert, find a text, or find a blog.

If anyone does decide to start such a group could you create a blog for the presentations and send me the link. It would be nice to print them out once you've been going for a couple of years so I could have that as a resource on my desk. Maybe at some point I'll spend a month a read through them all ;)

Benchmarking javascript with dropsecret.com

2009-05-04T14:32:00.000-07:00

Decrypting a message on dropsecret.com can take a noticeable amount of time. This is because there are a fair amount of operations involved in decrypting an rsa message. But there is also a noticeable amount of difference between the time it takes different browsers. So I added a simple benchmarking ability to dropsecret.

Simply go to your drop page on dropsecret (if you don't have one you can create one for free) and add the hash #benchmark to the url; ie

http://dropsecret.com/{drop page name}/#benchmark

(You may have to the page to get the new js file). Then decrypt one of your messages. The time it takes will appear the in the top left corner.

Some of my results in Windows 7 are:
IE 8: 61.147 seconds (after telling it not to abort the javascript several times)
Safari 3.2: 57.813 seconds (after telling it not to abort the javascript several times)
Opera 9.6: 37.426 seconds
Firefox 3.0: 27.759 seconds
Chrome 1.0: 1.616 seconds (no really, 1.616 seconds).

Of course your results will vary based on your machine speed and the message length. But now you can test out new browsers claims to have fancy fast js.

The trouble with confusing not observing with not existing

2009-04-28T13:11:00.000-07:00

It seems like many have taken the reasonable statement

we identified the swine flu in Mexico last week, then we started looking else where and found it there too

and made the leap to

the swine flu first appeared in Mexico last week and since then has traveled to other countries.

The problem with the second statement is it makes it appear like the swine flu is traveling much faster than we actually have evidence to support. While me may have only been finding swine flu in those who have recently been to Mexico that may be because we have only looked at the people who have recently been to Mexico.

The same flaw in reasoning leads to people thinking that autism is on the rise. If you look for something somewhere you are far more likely to find it there.

How to survive Swine Influenza

2009-04-28T10:45:00.000-07:00

Hey, I have my first guest blogger. A professor and my infectious disease expert... oh, and my father.

Michael A Noble MD FRCPC

Professor, Department of Pathology and Laboratory Medicine

University of British Columbia

Vancouver Canada

It’s springtime in the northern hemisphere once again; warming temperatures, clearing snow, cherry blossoms, and our annual reminder that influenza is not a winter time illness. Dry cough, head ache, fatigue lasting usually for about a week. If you are very young (under 5 years) or maturing (over 65) or have serious underlying illness, especially cancer, heart or lung disease, then things can be worse. Likely most will be unaware, some will get sick, a few will get very sick, and a very few will die.

But worse than influenza are the illnesses that come with it, in particular journalistic Alzheimer’s. It’s like every time is the first time. A time to use words like “epidemic” and “pandemic” and to bring out the pictures and stories of the great pandemic of 1918 (folks that is over 90 years ago!). And then there is the daily body count about how many died around the world yesterday, not counting those with old age, malaria, car accidents, drugs and alcohol, etc., etc, etc. It’s probably more likely to make you sicker than just catching the darn infection.

So this year it is from pigs somewhere in the world, and the first recorded manifestations are from Mexico. (This doesn’t mean it started in Mexico. It just means that is where the first people who died and had a sample sent to a special reference laboratory had lived).

Here is what you need to know to survive this years onslaught.

Don’t kiss or lick any Spanish speaking pigs.
Turn off your television.
Don’t buy or read a news paper.
Wash your hands regularly. Soap and water is fine, and no, you don’t need special soap.
Carry a small bottle of alcohol containing gel, preferably without perfume or coloring because they will be irritating. And don’t drink it.
Carry some tissues with you. If you cough or sneeze, use a tissue. Coughing or sneezing into your elbow is just disgusting.
Don’t go and buy or wear a mask, unless you like them as a fashion statement.
If you wake up with fatigue, dry hacking cough, headache and a fever, and you have not been out smoking and drinking all night, then you might have the flu. Stay home. If you get sicker, go to a clinic or hospital. There are pills that you can take, some by prescription, others, less proven, without.

And next year, you can go through it all, all over again.

Send passwords securely

2009-04-27T18:44:00.000-07:00

It has always bothered me that there has only been two ways to use encryption to send secure information around. Either you do it the hard way which means both the sender and the receiver have to use special tools and copy and paste long ugly base64 strings.

Or you can trust a 3rd party with your information and he will take care of all the encryption mess. But the whole point of encryption is you don't want a 3rd party to have your information. You should not have to trust your email provider with your secure communications.

So I created a website to solve this.

DropSecret is the easy and secure way to send passwords, credit card numbers, or any other sensitive information to those whom you trust.

How easy? To send a message you don't need to know any crypto information, have a password, or even have an account. To receive your messages all you need to keep track of is your pass phrase.

How secure? So secure that even with complete control of the server we don't know what your messages are. A drop secret page has a 1024 bit public key that the web browser will use to encrypt your messages. Then the only way to decrypt the message is to use your with your pass phrase.

And why should you trust me that I've implemented everything securely? You shouldn't. That's why I've open sourced the Drop secret encryption library. People should check it out and make sure there are no mistakes (or back doors for that matter). And if you like the library but don't like dropsecret then feel free to use the library make something new yourself.

Two bloggingheads worth listening to this week. on evolution and economics

2009-04-24T13:38:00.000-07:00

There were two bloggingheads this week that struck me as interesting. First there is a nice talk by two economists comparing modern thoughts on economics to classical to keynesianism to monetary policy to their own new ideas stemming out of the current economy. They bring up some points that I've been thinking about but haven't been able to express; like just because high interest rates stems off investment does this have to imply that low interest rates encourage it?

Then there is a discussion on evolution with Robert Wright (the owner of bloggingheads) and Joan Roughgarden (the author of The Genial Gene). Note that this is not a debate on whether Darwinism is true - that is accepted as a given. What this is is a debate on how much proof each theorized evolutionary mechanism requires, with Wright taking the position that evolutionary theories can't be proven with the same rigor as the rest of science so we should accept theories that sufficiently explains the data. Where as Roughgarden takes the position that there is no excuse for lowering the bar and that at the minimum you must come up with plausible competing theories and then apply rigor and statistics to determine which is more likely.

Roughgarden appears to be a controversial figure in many places on the web and for some reason receives a lot of animosity from blog commenters. I don't know if she has made other comments to earn this disrespect but in this conversation is both interesting and reasonable (and as far as I'm concerned she is also right).

In comparing these two conversations I wonder if I am engaging in an internal contradiction. From the first conversation I am sympathetic to the argument that applying statistical models to econometrics can be meaningless where as in the second conversation I am sympathetic to the argument that applying statistical models to evolutionary mechanisms is necessary.

Never shoot down in politics

2009-04-01T16:57:00.000-07:00

I just got a gmail ad for a BCNDP sponsored site that attacks votesmartbc.com - a site that attacks the BCNDP. Now while I did end up reading the BCNDP attack site I also ended up reading votesmartbc for the first time.

I think this ends up being a net loss for the BCNDP. I end up exposed to criticisms of the BCNDP I wouldn't have otherwise seen, and I end up exposed to criticisms of the critics. But I can't vote for or against the critics.

two partial fixes for the cufon select and copy problem

2009-03-22T16:43:00.000-07:00

Recently I've been spending Sundays playing with something new. Here's today's results.

I started playing with cufon today, which is a nice little javascript font renderer; ie a sIFR replacement for js. It has a big Achilles heal though: you can't select the text to copy it. So I experimented with a couple possible solutions here and here.

In the first case I've setup overlays so that when you click on a cufon rendered block of text you it transforms into the unrendered version until you click elsewhere. In the second experiment you can select and copy but you don't see your highlighting.

Both of these are just experiments and certainly not complete solutions (and they've only been tested in firefox). But perhaps they will give someone the inspiration to come up with a complete solution.

Improved smooth curves in javascript: now with more smoothiness

2009-03-12T14:05:00.000-07:00

I just updated the smoothcurve.js library at github (first blogging) because I had a few mathematical insights.

I decided to drop out any emphasis on doubly differentiable curves: those polynomials are still supported but they seem to offer very little extra. More importantly I added a 3rd method to determine the slopes at various points. I think the result is smoother curves.

Previously I was using a sort of heuristic to attempt to force the interpolation curves to fall strictly inside the rectangle defined by the data points. This is the method used in type 2 smoothing. Type 3 is actually guaranteed to remain with in this box (unless I made a mistake in my math). This is done by making sure the slope is always less than 3 times the slope of the diagonal of the rectangle.

Smooth data i is the curve generated with type i where type 1 is basically catmull-rom, type 2 is my first attempt, and type 3 is the new smoothing type.

Update: Added the ability to use the new smoothing technique without respecting local maxima and minima (which I call sanity).

First, here is an example with a small fixed data set to see the differences (or you can go to the page if the iframe doesn't work for you).

And here is the large set of random data (and here is the page).

Smooth curves in javascript: fast interpolation for large datasets

2009-03-03T12:27:00.000-08:00

The other day I was thinking about generating some data for some blog posts. Since I needed to procrastinate first I started by looking for how to plot smooth curves and I couldn't find a good solution. So here's my solution.

It generates graphs that are doubly differentiable (function is continuous as are the first and second derivatives). There are two smoothing types: the first is a little more flowing while the second attempts to respect the initial data as local ~~maximums~~ maxima and ~~minimums~~ minima. It's also fast enough to generate smooth data perceptually instantaneously with thousands of points.

I can write about how the math works in a future post if people are interested. These methods should not be used for interpolation for those interested in data mining or analysis. There are many other good algorithms for those uses.

And this is what the plots look like. The actual plotting is done by flot. Note that the data is generated randomly every time the page has reloaded. So I probably haven't seen the graph that you are currently looking at.

Update: I added the singly differentiable versions of this data. Looks pretty much like my previous plots. The type 1 singly differentiable plot is essentially catmull-rom to my understanding.

Update 2: This has been updated and blogged about here. You can find new examples there.

Famous mathematicians whose works have affected me

2009-02-20T14:09:00.000-08:00

I'm sure I'm leaving out a ton of obvious people but this is just a quick unordered list of famous mathematicians whose work I can recommend. With any luck I will evolve this over time. Hopefully give each their own post.

Pierre de Fermat

Carl Friedrich Gauss

Leonhard Euler

H.S.M. 'Donald' Coxeter

Benoît Mandelbrot

Serge Lang

Elwyn Berlekamp

Pafnuty Chebyshev

Joseph Fourier

John H. Conway

Évariste Galois

That's it for now. I know there are missing ones (names that are just out of reach) but I'll have to make another list for them later.

Making a drm free smart playlist in itunes

2009-02-11T00:32:00.000-08:00

Here is a trick I figured out when trying to find what music I own could be played on a linux box. If you go into itunes and create a smart playlist (command-option-n) then add the single rule "Kind :: does not contain :: protected" then you instantly get a playlist of your library with the drm'd music filtered out. To make this a little more useful you may want to add the additional rule "Podcast :: is false."

Frum auditions for Kristol's NYTimes column

2009-01-31T18:06:00.000-08:00

David Frum recently wrote "What's good for Rush Limbaugh is bad for Republicans" for the national post where he discusses Republican popularity and what they must do to improve it. You can get the gist of his argument from this quote:

Rush and Hannity and O’Reilly and Ann Coulter and the others have their place and their role. [...] The more the party allows them to become our public face, the more embattled and endangered the party becomes.

So we have an ex establishment Republican (he used to be a speech writer for Bush Sr.) writing an article suggesting his party move towards moderation and reject non-moderate influences. All this just after Bill Kristol leaves the nytimes. Can this be read as Frum throwing his hat in the ring to be Kristol's replacement? The article certainly sounds like if comes from sort of conservative the nytimes can be comfortable with. Even the sort of conservative who gets invited to have dinner with the president.

Mashing up dropbox and R

2009-01-26T22:24:00.000-08:00

This has to be one of my favourite things I've ever done with dropbox. I'm running a survey test in R on a whole bunch of data for a time series assignment (just running the test on everything like this is definitely the lazy way to do this, and I was always taught lazy is good in math). Since I was worried about the computer crashing while the test ran I'm outputting the test results to files. At the last moment I thought that it would be great to have this automatically backed up and accessible remotely. So I'm saving the output files to dropbox, with the result summary in the file name. In other words I am doing this:

>for(p in (2:3)){for(D1 in (0:1)){for(q in (2:15)){for(P in (0:1)){for(Q in (0:1)){
+ testmodel<-arima(x=tsphone, order=c(p,0,q), seasonal=list(order=c(P,D1+1,Q))); + print(p); print(q); print(P); print(D1+1); print(Q); print(testmodel); + save(testmodel, file=paste("./Dropbox/question2/", testmodel$aic,p,q,P,D1+1, Q, ".rdata")); + }}}}}

And since the smallest aic is probably the right model I can just sort the directory and see the best results at the top. I'm very happy about this.

Towers Perrin: an old business sticking its toe into the new web

2009-01-25T17:02:00.000-08:00

In my job search for an actuary job I've ended up looking at the websites of many insurance and HR corporations. The vast majority of them are modern and clean as I'm sure most of these companies have realized that their website is their main channel of external communication.

Amongst all of these sites I think it is worth looking at Towers Perrin as an outlier of what these companies are doing in using new web communication tools. Beyond a modern website they have a podcast (which I wish they would post more to), use twitter, and blog with some regularity.

It's not all that interesting when media companies use these tools -- it's starting to be surprising when a media company doesn't use these tools. But this is pretty far ahead of the curve when it comes to a non-media company. I would be interested to find out where the interest is coming from to reach out to the web like this. Is this someone high and central at Towers who is pushing this? Or am I just seeing a lot of action from a small group with in Towers which has been given the permission to try this as an experiment. And of course I would love to know what sort of rewards they are reaping from communicating in this space.