arbitrary user

Jonathan Endersby, Recovering Technologist

Archive for category Tech

Danny the capturer of the world.

Oct 13

Posted by arbitraryuser in Aggregate This, Observation, Philosophy, Tech | No Comments

Many years ago I worked at company that sold widgets. These widgets were very complicated and required lots of customisation. The company had developed a pretty large piece of software to help their sales people build complex widget quotes with lots of line items.

This company also had a big off the shelf enterprise accounting system that handled their real accounts.

I had worked at the company for almost 2 years as a software developer when one day I found myself sitting in the accounts department helping Danny with something unrelated. It was then that I learnt what Danny from Accounts actually did.

Every morning Danny would print out the previous days ‘accepted’ quotes from the quoting software resulting in a small pile of paper, one for each customer, with hundreds of line items, for every day. Then, using a ruler and pen to scratch out the lines, he would manually re-enter all of the customer data and their quote information, line item by line item, into the big accounting system. This process took him most of the day, sometimes more if business was good. He occasionally made mistakes that either cost the company lots of money or pissed off the customers.

As a software developer I knew that both systems ran off MSSQL databases. I knew that all the relevent information probably already existed to do the “job” programmatically. I knew that it would probably take a day or two to write a piece of software that did Danny’s job, perfectly every time, in a few milliseconds.

Danny had been doing that job for almost 6 years.

Since that day, whenever I start working with a new company, I try my best to meet everyone and get an idea for what they do and how they do it before I put my head down and start trying to solve any problems. That habit has served me well. In a team of ba/tech/strat/arch people I’m often the only one who knows how the accounts actually work, or how the stock is really procured, or what the weird hippies on the third floor do. (They’re always copywriters.)

But I’m not trying to pretend I have special powers. My point is that you can never assume that other people will have looked at problems like you do, with your knowledge-set. Most of the time other people won’t even see something like that as a “problem”. Danny’s boss never thought to question the process that admittedly pre-dated him. They all have no idea what SQL is and neither should they need to. It’s not their job. It’s yours. (Assuming you’re in a tech field)

What really excites me is how this kind of technology-discovery can be applied to people who traditionally live without the exposure to technology that we do. We now live in world where mobile phones can do things that sometimes even I think are quite magical (think SoundHound and Shazaam). I don’t know what “Danny the capturer of the world” situations exist in an under-resourced high school in a Soweto. I don’t know what efficiencies might just be waiting to be discovered in a clinic in Khayelitsha. I am however convinced that if a large corporate focused solely on profits with a really good, international, management team and a chartered accountant CFO all couldn’t spot that Danny was unintentionally wasting his time (and their money), then I can only imagine what amazing, albeit probably simple, tech-opportunities are waiting to be discovered in the “real” world.

I may not be ready to tackle the townships just yet, and I’m by no means assuming that there aren’t already smart people doing this kind of stuff, but I do look forward to one day being able to spend a few weeks immersed in the daily grind of a township school teacher or a minimum-wage worker, and maybe finding some way to bring a little bit of technological awesomeness and efficiency to their lives.

I know you’re wondering. I did write that software and Danny did need to click a button every morning and watch as the script whizzed by in less than a second, but he didn’t lose his job, instead he was able to move on to tackling more challenging things that actually needed his accounting skills. Everyone’s a winner.

Pystack and Djangoverflow

May 26

Posted by arbitraryuser in Tech, hacking | No Comments

A while back I was chatting with Brad and said “Someone should create a Twitter feed of popular Stack Overflow questions“.  I think he said “Yes, someone should“.

Later that night I created @pystack and @djangoverflow.

Sometimes you just have to code.

Content and delivery.

Feb 2

Posted by arbitraryuser in Blogroll, Life, Observation, Tech | 1 Comment

Recently a friend who’s in the magazine industry was complaining about how their company (who is a very large media company) continually cuts the budget of the magazine people (ie. Those who actually produce content) while at the same time spending gob-loads of money on their “Online” and “Mobile” people. I’m hearing stories of games rooms and guys walking down the passage with an iPad in one hand and a Macbook Pro in the other, while just down the passage there are magazine teams running on 10 year old macs.

The print-media industry is no doubt floundering. Seeing demand for its product dropping by significant numbers every year (We’re talking overall sales figures around 20% of what they were 10 years ago) while ad-sales are becoming more and more brutal due to the “global economy” and, probably more scarily, losing ad sales to online channels. Fewer people want to buy their stuff and they’re making less and less off that reduced number.

So you can imagine the kind of pressure the industry is in and how incredibly easy it would be to come to the very foolish conclusion that the correct remedy is to spend those gob-loads on “Online” or “Mobile” to the detriment of the content producers.

My father was a printer, technically an offset lithography “machine minder”. Practically that meant that he was badly paid, worked long shifts, went to work in blue overalls and came home covered in ink. The work was tough. You needed to have an expert eye, understand some of the chemistry, have delicate hands and be able to perform running repairs on the printers. We’re talking about giant room sized machines and having the ability to hear that the third roller bearing on the transfer shaft dingle dangle needed oil in the next 30 minutes or the machine would fail. (I’m paraphrasing)

The reason my dad was badly paid even though his job required so much skill was because lithography was an old technology. The mystique had been removed from the process about 100 years earlier and the machines looked after themselves just enough to be able to have an unskilled worker become fully skilled in 3 years of on the job training.

The technology was mature and there was solid competition in the market. That drove the printing prices down, which pushed the salaries down, which meant that eventually it was only slightly more attractive as a career than something like panel beating.

In the last 30 years printing has evolved to the point where the machines are easier to use, faster and even more reliable. Instead of hiring one or two guys per machine you can now have a few roaming engineers for an entire factory of printers. Putting ink on paper has never been cheaper.

My father moved to the publishing world about 30 years ago and has been wearing chinos to work ever since… Though I’m pretty sure he would still prefer to deal with machines than colleagues.

The cost and skill involved in delivering content will always drop. Technology takes care of that. However we will never have Artificial Intelligence that can honestly go to Darling and write about an Evita Bezuidenhout show, take photographs of the flowers in the Karoo, write about swimming with dolphins on a cool Sunday morning or editorialise about crappy Egyptian presidents.

100 years ago quality content made money… That is still the situation and it is unlikely to ever change. How that content is delivered should never become more important than the content itself.

You might be able to wow people with your swanky iPad application with annoying faux-turning-pages animations, but eventually the technology will mature and everyone will have swanky iPad apps. The cost involved in building those apps will drop and the big boys will be consistently competing against small, leaner, startup content producers. And as if to amplify the situation the technology is changing much much faster. It took hundreds of years to get the cost of printing on paper so low that we could afford to print a daily newspaper and sell it to the masses. The cost of producing an Ipad app halves every six month and, as the technology evolves, it becomes trivially easy for anyone with some good ideas and camera to create something that other people want… and god forbid, would actually pay money for.

So, if you happen to be the CEO of some big ass media giant, spare a thought for Gutenberg and then Google “ios and android development frameworks” before deciding not to buy your content producers some decent computers. You could even do it on your iPad.

Standard Bank, Information Theft and Apathy.

Apr 8

Posted by arbitraryuser in Aggregate This, Rants, Service, Tech | 3 Comments

We place our trust in the organisations we deal with on a daily basis. Many of these organisations have access to large amounts of our personal information. We assume that there are systems in place to protect consumers against the theft of that information by staff members. We assume that if a staff member did steal data, that the bank would treat the matter with urgency. We assume that the bank would press charges against the individual and try and recover all the stolen information.

We assume incorrectly.

About two weeks ago I was called by a guy called Riaan Geldenhuys. He is supposedly a financial planner and wanted to meet up to discuss policies etc. I was intrigued as to where he got my details from so I played along and then eventually asked him where he had found my number. His answer was, in context, quite shocking. He was an ex-Standard Bank employee and had a “list” from when he worked at Standard Bank.

If we ignore for a second the absolute idiocy of this guy actually telling me that, I was obviously left wondering what other information he had on me… I told him that I would report him and that I hoped the bank took him to court. He said he was “sorry”.

In a world where information theft is rife, bank fraud (using information theft) is all too common and everyone is legally married to at least two people they’ve never met, I would have thought that my reporting of this matter to the bank would have resulted in an urgent phone call from the fraud division, asking me for all the information and then a subsequent call a few days later telling me that this Riaan guy was going to court. But no. Typical of their lackadaisical concern for our privacy, the banks auto-responded and then a week later told me they were “looking into the matter”. I let them know that “looking into the matter” wasn’t good enough for me… I wanted to know that at the very least my, and their other customer’s, stolen information had been retrieved. Another week goes by and still, nothing but auto-responders and “we’re looking into it”s.

Eventually, in an incredibly bitter fit of resentment I succumbed and wrote about the issue on HelloPeter, a site I usually avoid because of their reputation of extorting businesses who can’t afford to pay to defend themselves.

Surprise, in a few hours I had a phone call. Now I was told I needed to go into a branch and fill in a form. When I made it clear that the chances of me doing so were similar to, well, just about anything very very unlikely, they then realised that they didn’t actually need me to go into the branch… With promises of swift action I wrote another long email, detailing the entire saga and hoping that it would finally result in some form of action.

Then this morning I am asked to email them either my bank account number or my ID number. The irony of me sending either over unencrypted email to the very bank that breached that information in the first place seemed to be lost on this new individual I was dealing with. Never mind the fact that I had given Standard Bank my account number numerous times, and never mind the fact that I am the only Jonathan Endersby in the country and never mind that really, my bank account has nothing to do with them investigating this Riaan Geldenhuys guy because lets face it, their internal audit logs are obviously quite sloppy, so assuming that they would specifically find detail that Riaan Geldenhuis had looked at my account is, well, optimistic.

To complicate (or obscure) matters, the bank (or at least Clinton who works in their Fraud Division) insists that the only way that anyone would have gotten access to my information would have been via an auditable system which has logs. I explained the retardedness of that statement, detailing how the emails I’ve sent with my information were obviously not tracked because they are apparently lost. Next I asked whether all their email is also linked to this magical system, because I’ve worked in a few banks and I know that the sales teams are big fans of Excel spreadsheets generated the Sales Manager and then emailed to pretty much everyone, including the tea-ambassadorial staff (corporate refreshment executives?). I suspect it is probably one of these lists that was stolen. Anyway, according to SBSA, if they can’t find access logs for Riaan Geldenhuys, “there is nothing we can do”.

Clinton also said that he was looking into whether Riaan Geldenhuys, financial planner guy, was ever employed by Standard Bank. He said they were investigating whether he was ever employed by Standard Bank by speaking to the division he worked in. Yes, that is exactly what he said. I almost shat my pants thinking that this guy is investigating fraud for Standard Bank. I’d like to assume he simply meant that they were speaking to HR, but I have suspicions.

To cut a long story short, I’ve lost hope that the bank will ever take this matter seriously and pretty much assuming that they have better things to do, like close at 3pm, than investigate information theft… I mean, it’s possibly only my id number, salary, home address, mother’s maiden name, signature, phone numbers, previous places of employment, information about the various insurance companies I use, who I invest with, previous addresses and all the account numbers for the various services I use…. I mean, you can’t do much with that information, can you?

There is no spoon – The challenge of unlimited bandwidth in a limited world.

Mar 31

Posted by arbitraryuser in Aggregate This, Service, Tech | 11 Comments

Change is constant. With increased international capacity it was inevitable that ISPs would eventually enter a price war. It was MWEB, a traditionally not-so-forward-thinking ISP, who shot first.

Uncapped internet for a price that didn’t seem insane – Terms and Conditions apply... It didn’t take long (a few minutes actually) before the nerds were frothing at the mouth over what seemed to be overly-burdensome (and in some cases just-plain-stupid) regulations. Rules like “No unattended downloading” being one of them… while in principle most people understood the ethos, the unfortunate reality is that rules shouldn’t be _made_ to be broken… and telling an old granny she can’t go make a cup of tea while her email downloads is simply not intelligent.

The problem is simple. Internet Service Providers have a limited resource and they are selling it on as an unlimited resource… It’s the all-you-can-eat ribs special, only in a digital world, where the limit to how much you can eat is simply a question of how big your hard drive is.

Most of the nerdosphere understood that ISP’s would have to enforce some limitations, and in fact, most ISPs worldwide have some form of Acceptable Usage Policy. The difference being that the kind of numbers that constitute abuse are generally in the range of hundreds of gigabytes/terabytes per month, and then only after consecutive months of “abuse”.

The problem in SA is that the business model is really hard to get right because it revolves around a number of unknowns:
1. What can we offer that’s good enough to a) Attract customers. b) Be called uncapped. c) Not piss off the nerdosphere. ?
2. How many customers can we sell this to?
3. What will the average usage of those customers be? (Ubernerds download a lot more than your Granny)
4. If we scale up operations because of a surge of new customers, how can we be sure those customers will hang around to support the increased running costs?

Additionally, ISPs are obviously terrified to not enter the market because not having an uncapped option will inevitably mean losing pretty much every customer who isn’t living under a rock.

So, possibly with a fair dose of fear and trepidation, a number of other ISPs quickly entered the market with their own offerings, all clambering to try and get that business model right.

Some ISPs even appear to have decided to start selling the product before they figured out what that business model would be. A bold move that cost the likes of Afrihost a fair amount of pain when they realised they needed to implement a soft cap (they call it something else) at 60gb. That 60gb number wasn’t anywhere on their website because it appears to have not existed when they launched… it was only after seeing the real usage numbers that they realised they needed to implement some additional limits. (After downloading 60gb your connection is throttled, and then once you hit 120 it’s throttled further etc etc)

So we come to what is really the crux of this debate. What is uncapped? Currently the uncapped market is unregulated and very unstable. The rules are changing on an almost daily basis and pretty much anyone can offer anything and call it uncapped. Someone could have a product that calls itself “uncapped” but that limits you to 1kbps after the first megabyte. This is not good for consumers.

The market is in need of a lot more transparency or a regulator. There are really only two groups that could play the role of regulator: The Advertising Standards Association and the Internet Service Providers Association. I’m ignoring ICASA for obvious, incompetent and toothless, reasons.

The ASA unfortunately doesn’t have the knowledge to regulate such a highly complex industry and any attempts to do so would probably have very negative effects for all involved.

ISPA on the other hand does have the know-how but hasn’t publicly said anything about the matter. All of the ISPs currently offering Uncapped ADSL are ISPA members. I think the only reasonable solution is for ISPA to get a bunch of its members together and lock them in a room until they can all agree on what the minimum provision for an uncapped account should be. This would need to be measurable limits and not warm-and-fluffy, open to interpretation, language. They may even decide that calling these sorts of accounts “uncapped” is dishonest, perhaps it should just be called something like “Managed Cap 60″ etc.

I look forward to the day that we have true uncapped internet in this country and I salute those ISPs who are trying their best to bring us closer to true uncapped internet. They are brave businesses operating in an increasingly brutal space.

Most importantly we need the ISPs to be honest about what they’re selling. If they’re selling something that has graduated throttling (like Afrihost is doing) they need to say so before they take the customers money. Afrihost doesn’t currently say this on their website, but their CEO has published (very bravely and honestly) the planned (and he understandably pointed out that it was plan that might change) approach on the mybroadband forums. I’m sure that this info will make it onto their website as soon as the dust settles.

Publishing the exact structure/behaviour of their uncapped product is a brave move that hopefully will force other ISPs to do the same. It’s only when all ISPs are showing their hands that consumers will be able to make an informed decision.

Quick and Simple Server SMTP

Sep 15

Posted by arbitraryuser in Linux, Tech, hacking | 3 Comments

I have a number of servers that I look after in various places on the intertubes. I like to have things like MDADM (Linux software RAID manager) be able to mail me when the something goes wrong like a disk dies etc.

Some of these machines are in places without reliable SMTP servers for me to send mail through and I’ve tried running my own postfix and delivering the mail directly, but invariably I run into situations where the servers that I’m trying to deliver mail to don’t like DSL IPs… and not getting a mail about a dead disk is kinda a big issue.

I also don’t trust a lot of ISP’s SMTP, and some of my servers move around, so one day it’ll be behind a DSL IP and the next behind a Verizon IP (where it can’t talk to smtp.dslprovider.net etc).

My solution is quite simple, use google. (This guide is for Ubuntu but I’m sure you’ll figure it out with other distros)

  1. Create a gmail account for monitoring. I do this because I don’t want my gmail password floating around in plaintext on various machines.
  2. Install the ca-certificates package

    $ sudo aptitude install ca-certificates
    $ sudo update-ca-certificates

  3. Install msmtp

    $ sudo apt-get install msmtp

  4. Configure msmtp

    $ sudo vim /etc/msmtprc

    Set it to something like

    account gmail
    host smtp.gmail.com
    from myemailaddress@gmail.com
    auth on
    tls on
    tls_trust_file /etc/ssl/certs/ca-certificates.crt
    user notifyemailaddress@gmail.com
    password mys3cr3tp455w0rd
    port 587

    account default : gmail

  5. Create a sendmail simlink

    $ sudo ln -s /usr/bin/msmtp /usr/sbin/sendmail

  6. Run a test

    $ echo “This is a an awesome test email” | msmtp youremail@domain.com

  7. If you want mdadm to mail you when something goes wrong

    $ sudo vim /etc/mdadm/mdadm.conf

    and put your email address on the line that reads something like

    MAILADDR youremail@domain.com

  8. And then run a mdadm test by running

    $ sudo mdadm –monitor –scan –test –oneshot

  9. If everything is working according to plan you should receive an email. You can now rest assured that any future MDADM issues will get to you.

Visualising the Interest Rate

Mar 24

Posted by arbitraryuser in Aggregate This, Property, Tech | No Comments

I though it might be interesting to try and graph the Reserve Bank’s prime rate data… It goes back a long way. I used Python to scrape and collate the data and PyCha to generate the graph.

UPDATE: I’ve replaced my graphs with new versions made by Russell who corrected my original code by interpolating the data correctly over the y axis.

This is the narrow version.

And this is the wide version (click to download the actual 10000px wide png)

Interestingly enough, todays rate cut *was* on that page earlier today, but now I see it’s gone… so I inserted it manually ;)

There is no cure for stupidity.

Feb 11

Posted by arbitraryuser in Aggregate This, Tech | 5 Comments

A while ago I blogged about a weird comment I had received on one of my blog posts.

In summary, there is an SEO company called SEO Results (aka BizSearch, aka NetAge) that gets its staff to trawl blogs and write comments with the Author URL set to the url of one of their SEO clients.

Author : PMM (IP: 165.146.34.239 , dsl-146-34-239.telkomadsl.co.za)
E-mail : kim@bizsearch.co.za
URL : http://www.pmmproperties.co.za
Comment:
Wow what a difference it looks fantastic, great job done

One would think that after the first run in I had with these spammers they would have avoided my blog?

Anyway, to make sure it’s clear: SEO Results are spammers and black hat SEO idiots… Using them is likely to get you bad mouthed on the internet (like this) and perhaps worse, blacklisted on google.

Eye Witness News (ewn.co.za) has a few issues.

Jan 20

Posted by arbitraryuser in Aggregate This, Tech | 5 Comments

First let me say that I like the idea of a new, fresh news site… EWN could quickly become a serious player in the news arena, but before they do so they’re going to need to fix a few issues.

I sent an email listing some of these issues to the Primedia team. I know it got there because people who know people said there was some flapping and urgent updating that happened as a result of the email… However, I’m yet to get any form of reply whatsoever… which I think is just rude.

(update: A few things (like the comments about Mandela) have been fixed, but the overwhelming majority is still as it was when I wrote this list a few days ago. The site however seems to be suffering from lots and lots of timeouts now.)

This list is by no means exhaustive…

1. You need to add a DNS record for ewn.co.za (so that http://ewn.co.za actually works)

2. You need to add RSS, preferably ATOM, with a number of sub feeds, geographic locality etc.

3. You need to remove your stupid comments from your html source… not only is it dumb, but people WILL take offence.

<!–<li><a href=”#”>Mandela Gives Birth to a Gorilla </a><span class=”timeadded”>2&nbsp;days&nbsp;ago </span> </li><li><a href=”#”>Prengant Child attacks Mandela</a><span class=”timeadded”>3&nbsp;days&nbsp;ago </span></li><li><a href=”#”>Tourists Can’t Give Enough Birth </a><span class=”timeadded”>1&nbsp;day&nbsp;ago&nbsp;</span></li>–>

etc

4. You need to make sure all your templates actually work… for instance this one is a little too concise –
http://www.ewn.co.za/story.aspx?id=4013

5. You need to protect yourself from SQL injection and handle any attempts gracefully.
ie. http://www.ewn.co.za/articleprog.aspx?id=40%2709

6. You should probably consider looking into better urls for your articles, specifically for SEO purposes.

7. You should also probably add meta descriptions (and possibly tags) to your article pages. This will help display relevant content in search engine results.

8. Your pages do not even come close to validating XHTML transitional.

9. You need a mobile version! This is easy to implement!

10. That logo… It’s very 90′s.

11. Bonus Tip: One of my biggest gripes with the other news sites is how they never allow you to view larger versions of their images. Implementing Lightbox2 over you existing site will be easy and help
differentiate yourselves from the other players.

12. Your site search is broken in Firefox and Safari and is unstable in IE6 and 7.

13. Your server errors (timeouts etc) need to be handled more gracefully. At the moment your site displays the default .NET error pages, which is something that only the developers should be seeing.

14. Your comment form gives no indication that it hasn’t submitted due to invalid data. This will confuse users.

15. Besides the SQL Injection issues, users who search for any string that contains an apostrophe will be greeted by a rather ugly error page. Try search for o’grady.

16. You need to remove all your test data from your database. http://www.ewn.co.z/articleprog.aspx?id=183 etc

17. You should add a clearfix after your pull-out-quote on your article pages. This will ensure that articles that start with single character words like “A” don’t end up displaying the first character to the right of the pull-out with the rest of the article below the pull-out. See http://www.ewn.co.za/articleprog.aspx?id=4021

18. Your logo should be a link to your landing page. This has become a web standard and a lot of users will expect it to do so.

19. You should sanitise your article source before your editors submit it so that you don’t end up with styling imported from MS Word which can break your layout. ie. 

<p class=”MsoNormal” style=”MARGIN: 0cm 0cm 0pt; LINE-HEIGHT: 12pt; tab-stops: 18.0pt 91.6pt 137.4pt 183.2pt 229.0pt 274.8pt 320.6pt 366.4pt 412.2pt 458.0pt 503.8pt 549.6pt 595.4pt 641.2pt 687.0pt 732.8pt”>

Not only is it ugly but it will repeatedly break your validation.

eg. on http://www.ewn.co.za/articleprog.aspx?id=4033

20. While it’s debatable whether this is a true bug, there is a fair amount of functionality on your site that is broken when the user disables javascript.

21. As I’m browsing the site I am hitting a lot of timeouts. This indicates that your server is probably struggling. Most likely due to bad coding and/or a database that isn’t tuned properly.

22. Your cache control is not good. You should probably add far future expire headers to all your static resources. This will speed up the site for regular users. Also, combine and gzip your js. This will also decrease load on your site and help with all the timeouts.

Girls and XHTML Validation

Jan 13

Posted by arbitraryuser in Aggregate This, Tech | 12 Comments

If you’re ever debating whether or not something is sexist, change the gender statement into a racial one and see how it fares…

ie. (taken from the intertubes)

Lucy
We don’t know a whole lot about Lucy, except that she’s one of the few females on the planet who can hold a conversation about search engine algorithm changes and validating XHTML pages.

Changes to:

Sipho
We don’t know a whole lot about Sipho, except that he’s one of the few black people on the planet who can hold a conversation about search engine algorithm changes and validating XHTML pages.