Like I assume millions of other people, I’ve been unable to make a payment with PayPal for over 8 hours now. The following error message appear on the PayPal website when trying to make a payment with a credit card…

The PayPal site is currently experiencing technical difficulties with our credit card processor. We are working to solve this problem as quickly as possible. If you would like to use your credit card, please return to the PayPal website later to complete your transaction. We apologise for any inconvenience this may cause.

Quite a concern considering eBay is now making it mandatory for all auctions to include PayPal as a means of payment. And as of July this goes one step further with PayPal being the only option to pay an eBay seller.

While using Commonwealth Bank’s online bank NetBank last week I transferred over $1000 on to my Credit Card. I didn’t owe anything on this credit card I just wanted the funds on there to use… Since then I’d noticed my Available balance was on top $2000 of my limit…

Really confused thinking the bank had screwed up and increased my limit without my authorisation I gave them a call. Just off the phone and I’ve found out they currently have a sporadic error occurring on NetBank.

Basically some transfers being made between accounts are registering the deposit twice, and the withdrawal once!! I.e. I was withdrawing $1000 and depositing it on my card, but it was registering 2 deposits.. so $2000 in total… CHING CHING!

The phone operator corrected the error despite me insisting it be left how it was He also said the problem would have been corrected in a couple of days anyway - but I wonder whether it really would have?

This is just one VERY good example of why you should build thorough logging in to your web applications for auditing purposes. Just think the amount of money this bank would be losing is they hadn’t!! Or perhaps are if they are relying on fools like me to point out their mistakes.

This raises an age old question which will likely be debated for many years to come. Ultimately both methods of storage have their benefits and costs.

Storing images on the file system has a marginally faster retrieval rate, thanks to web and proxy servers being good at what they do.

Storing images in a database allows for all of your data to be central stored which is more portable, and easy to replicate. This solution would likely also be easier for taking a point-in-time backup with referential integrity.

Which option you choose would really depend on the type application you’re building in my opinion.

So if you’re building an application with a moderately sized amount of image data, and moderate amount of traffic using a database would be okay as the benefits outway the cost. However if you’re building something like flickr with large amounts of data and high traffic, using the file system would be the advised approach.

I’ve also heard of a combined solution that could provide the best of both world. This is storing your images in the database to gain the benefits there, but also use filesystem caching of these to obtain the performance benefits.

For a senario of a small photo storage site with 2 Gig of images, I would recommend the filesystem approach or consider attempting the combined solution. Although at only 2 Gig either approach would be fine… but we need to allow for some growth, it could boom right?

Some tips for getting the best performance out of the filesystem:

• Limit the number of images in any one directory (or suffer performance loss)
• Include not only an image identifier in the filename, but also a secret code (to prevent discovering files)

See the following website has some great information on flickr:

http://www.highscalability.com/flickr-architecture

Additionally there is this presentation on scalable web architechure:

http://www.slideshare.net/techdude/scalable-web-architectures-common-patterns-and-approaches

Zend debugger is basically the server side component that is used by Zend Studio. You need this to do your remote debugging!!

Normally you’d find this included in an installation of Zend Core or Zend Platform. However we aren’t using Zend’s apache php bundle so this little gem has to be loaded…

It’s hard to find on the net and not well advertised - I imagine because of Zend’s preference that you’d start using Zend Core.. Which looks great mind you but is missing some vital extensions some of our projects are using..

http://downloads.zend.com/pdt/server-debugger/

My problem is I and working on a web interface which runs some limit shell command. Some of these normally require root access… And I don’t want to do something silly like force apache to run as root now do I?

Provided you have sudo installed (like most distro’s) the following is a good solution I came across!

Update your sudoer config (mines at /etc/sudoers) so your apache user can run the required command.For example:

Cmnd_Alias TOOLS=/usr/sbin/yourcommand,/usr/sbin/anotherone
www-data ALL=NOPASSWD: TOOLS

Then in your PHP you would execute the command like so:

exec("/usr/bin/sudo /usr/sbin/yourcommand");

If anyone can suggest a better method I’d love to hear!!

My goal was more complex than what’s described here in, but I wanted to share a simple function for returning the links in some HTML (now that I know what I’m doing)… Hopefully someone finds this useful, it was a common question in forums I noticed.

Regular expressions are a power tool for working with strings. PHP provides support for a couple of different types but I’m using preg (aka the Perl compatible one).

The regular expression I put together for this was:

/<a\s[^>]*href=”(?P<href>[^"]*)”\s[^>]*>(?P<name>.*)<\/a>/si

What this means is:

• / - perl regular expression patterns are enclosed in forward slashes (this is the opening one)
• <a - is satisfied literally (the open of the html a tag)
• \s - is a single whitespace character (includes line breaks etc)
• [^>]* - satisfied by any characters except >, this can be satisfied zero - many times (allows for anything else inside the html a tag)
  • [ ] - a charter class
  • ^ - except the following
  • > - is satisfied literally
  • * - the charter class can occur zero of many times
• href=” - is satisfied literally
• (?P<href>[^"]*) - match and return as ‘href’ - any characters except “, this can be satisfied zero - many times (gets everything inside the href attribute)
  • ( ) - match and return
  • ?P<href> - nominate the name we’ll return it as ‘href’ could be anything you like!
  • [^"]* - satisfied by any characters except “, this can be satisfied zero - many times
• > - is satisfied literally (the close of the html a tag)
• (?P<name>.*) - match and return as name - any character, this can be satisfied zero - many times (gets everything inside the a tag)
  • ( ) - match and return
  • ?P<name> - nominate the name we’ll return it as ‘name’.
  • .* - satisfied by any character, this can be satisfied zero - many times
• <\/a> - is satisfied literally (but we’re escaping the forward slash we don’t want to end up pattern here)
• / - now we want to end our pattern!
• si - the trailing s and i are modifiers to change the way the expression is interpreted
  • s - means the . we’ve used can also represent line breaks (normally it doesn’t)
  • i - means the entire thing is case insensitive!

A PHP function using this might look like so:

private function getLinks($responseBody){       
    $_regexp = '/<a\s[^>]*href="(?P<href>[^"]*)"\s[^>]*>(?P<name>.*)<\/a>/si';
    preg_match_all($_regexp, $responseBody, $matches);
 
    $i = 0;
    foreach($matches['name'] as $name) {
        $links[$i]['name'] = trim($name);
        $i++;
    }
 
    $i = 0;
    foreach($matches['href'] as $href) {
        $links[$i]['href'] = $href;
        $i++;
    }
 
    return $links;   
}

Issues with this regular expression I know I haven’t address are:

• You’re link may not be text, it could be an image or anything!
• Not everyone using double quotes for their attributes.
• Browsers support sloppy HTML this experession doesn’t! E.g. <a href = /link/>

Any corrections or feedback would be pleased to hear from you!

Trac is a popular open source issues management system written in Python. It also features a wiki and subversion browser. There are also countless plugins that have been created for it some of which I’m surprised aren’t included in the base install.

I’m installing this on a Ubuntu server, so you may need to find installation instructions to suit your distribution! To install the packages:

apt-get install trac libapache2-mod-python

Now to setup our first Trac project:

mkdir /var/trac
trac-admin /var/trac/example initenv

The last command will step you through some questions:

Project name - Being creative I named mine “Example“.
Database connection string - I hit enter using the default.
Repository type - I hit enter using the default as mine is SVN.
Path to repository - My repository for this example was /var/svn/example.

Next you’ll need to make sure your web server can access the files:

chown -R www-data /var/trac/

There are two ways of serving Trac through your web server one using CGI the other using the mod_python which I’ll be using. Include the following in your apache config:

 <Location /trac>
     SetHandler mod_python
     PythonHandler trac.web.modpython_frontend
     PythonOption TracEnvParentDir /var/trac
     PythonOption TracUriRoot /trac
 </Location>

This doesn’t include any security as you’ll see in most install tutorials. This is because I plan on installing a plugin to make use of a web based form log in instead.

If you restart apache then open http://ip-address/trac/ in your web browser you should see you list of Trac projects. Log in won’t work yet so we’ll continue…

Installing some Trac plugins

Now we’ll install some plugins you really can’t live without! We’ll start with WebAdmin which an interface for trac-admin command line utility. This would normally be used for adding component names to your projects - something you really should have to drop to a shell for! I’ll also install the AccountManagerPlugin while I’m at it…

First we need to configure the server to be able to install python eggs! This is how all of the Trac plugins are packaged…

apt-get install python-dev
cd /tmp
wget http://peak.telecommunity.com/dist/ez_setup.py
python ez_setup.py

Now that thats done lets get the plugins installed! Please note you may need to refer to the plugin websites to ensure you’re using the correct repository for your version of Trac.

easy_install http://svn.edgewall.com/repos/trac/sandbox/webadmin/
easy_install http://trac-hacks.org/svn/accountmanagerplugin/0.10

Now that these are installed we need to configure our Trac project to make use of them!

Open the trac.ini file inside under the /conf/ directory of your Trac project. Mine is located at /var/trac/example/conf/trac.ini

[components]
trac.web.auth.loginmodule = disabled
webadmin.* = enabled
acct_mgr.* = enabled
acct_mgr.web_ui.RegistrationModule = disabled
 
[account-manager]
password_format = htpasswd
password_store = HtPasswdStore
password_file = /var/trac/trac.htpasswd

We should setup that password file and create our first account:

htpasswd -c /var/trac/trac.htpasswd admin
trac-admin /var/trac/example permission add admin TRAC_ADMIN

Some plugins also use caching if you don’t follow this next step you’ll run in to some nasty errors (Permission denied: ‘/root/.python-eggs’) ! This is because the web server can’t write to the location they use by default… So:

mkdir /tmp/trac-cache
chown www-data /tmp/trac-cache/

Then include this line in your apache config along with the others for the /trac directory.

SetEnv PYTHON_EGG_CACHE /tmp/trac-cache

Then restart apache and cross your fingers!

Update!

After my success with this install I started looking at Trac-Hacks through all of the other plugins and also decided to install the following (I’ll update this as I find others):

• CustomFieldAdminPlugin

I’m involved in the development of a large web application (using Zend Framework) with many different types of entities inside! Some functions performed on these entities should be accessible to some users but not others… To make things more interesting users can assign other users with rights to these entities…

We’re going for a hands off user administration model where users register themselves, create things themselves, and give other users access to them.. THEMSELVES!

This article is Part 1 of many as I design and create a Zend Controller Plugin designed in simple terms to check if the user is allow to do whatever the hell they’re trying to do… I’d like to reuse it forever and a day though, so we need more functionality and flexibility…. My list of requirements goes on:

• Simplified everything - Can we use 1 line to kick off all ACL checks?
  No one likes having to include the ACL checking at the top of everything, and worse yet implicitly define what it is the page does, or who should access it… So we’ll require none of this!!
• Anonymous users should not be discriminated against!
  Anonymous users should be treated just like any other authenticated user - their access should be checked using the same process, and stored in the same manner! Examples of benefits:
  • Simplifies management the ‘anonymous user’ by using a normal role.
  • Logging (e.g. Ann performed this action) and metadata (e.g. updated by Ann) functions don’t need to cater for the irregularity of an ‘unknown’ user.
  • Explicit access to entities can checked/stored in the same way as normal users. For example - I want to make my Social networking profile publically viewable.

• Users should have generic roles!
  This means even though some users may not have implicit access to some entities, they can still have access! This is a more typically found smaller applications so definitely needs to be included.. Some example usage scenarios are:
  • Administrators - Access to everything!
  • Support staff - Application support staff may need to see ’stuff’; manage user accounts.
  • Moderators/Editors - Can approve anything/Can edit anything.
  • Anonymous - Allowed to see log in page.

• Users might have explicit privileges to supercede generic ones!
  This provides a way for use to either grant or restrict access to specific entities regardless of the users generic role. An example scenarios:
  • “Support staff” should b restricted from managing Administer accounts.
  • A user by default can’t edit ‘comments’, but can edit their own.

• Actions can depend on actions too!
  If a user has permission to perform a certain action, there may be other actions they should also have access to automatically. Example scenarios:
  • The ‘Register user’ action uses another action for AJAX validatio
  • The ‘Latest news’ action has an equivalent RSS/ATOM feed through another action

• Required ID checking for actions (soo not typically in scope, I know!)
  I need to explore this idea a little further (I’m not sure it will make the cut yet)… This typically wouldn’t be included in scope for a access control class! BUT… In this ACL model we’ll allow for checking if a user has access an perform an action on a specific entity. And we need to know the entity’s ID and what the entity is right?… So it seems only logically to store what actions REQUIRE what types of entity ID’s… Maybe we’ll go so far as to check they exist?

Assumptions

I’ll never have access requirements more specific than the action being requested!

Basically my ACL will define a users permission (ie. can access OR can’t access) to a specific action and its relationship with an entity if applicable. It will not be able to say a user can only partially see/run the action.

This will likely however still be achievable within this application itself. But it’s definitely outside of the scope of this particular Controller Plugin.

By ‘action’ I mean the standard route in Zend Framework (ie. Module/Controller/Action). So we’ll assume I’ll always be using it…

End note

Obviously every project always has unique requirements so I don’t anticipate what I’m working on will suit EVERYONE… But considering I’m designing this for maximum reused I would very much appreciate hearing the many weird and unique access control requirements you’ve encountered.

While I’m working on it I would like to incorporate the anything that is realistically reusable…

Currently Zend_Auth won’t work if you’re using a Micrsoft SQL Server database for storing your account credentials.

This is because of a bug in the \Zend\Auth\Adapter\DbTable.php specifically in the authenticate() function. The SQL Statement it generates is not MS SQL friendly:

SELECT "users".*, "credential" = 'mypass' AS zend_auth_credential_match
FROM "users"
WHERE ("identity" = 'me')

Consequently causing the following error:

Incorrect syntax near the keyword 'AS'.

The good news is the code below can be used as a replacement in this function until the Zend Framework team get a chance to fix it themselves. It has been tested in MS SQL 2005 but I imagine it should work well in another DB (but test this yourself and comment back!).

// build credential expression
if (empty($this->_credentialTreatment) || (strpos($this->_credentialTreatment, "?") === false)) {
    $this->_credentialTreatment = '?';
}
 
$credentialExpression = new Zend_Db_Expr(
    $this->_zendDb->quoteInto('(CASE WHEN '
        . $this->_zendDb->quoteIdentifier($this->_credentialColumn)
        . '=' . $this->_credentialTreatment, $this->_credential)
        . ' THEN 1 ELSE 0 END) '
        . ' AS ' . $this->_zendDb->quoteIdentifier('zend_auth_credential_match'));
 
// get select
$dbSelect = $this->_zendDb->select();
$dbSelect->from($this->_tableName, array('*', $credentialExpression))
         ->where($this->_zendDb->quoteIdentifier($this->_identityColumn) . ' = ?', $this->_identity);

The code above generates the following MS SQL friendly SQL statement:

SELECT "users".*, CASE WHEN "credential" = 'mypass' THEN 1 ELSE 0 END AS zend_auth_credential_match
FROM "users"
WHERE ("identity" = 'me')

There is an issue open with the team if you’re interested in reading it (and please vote for it to be resolved).

There’s a lot of buzz with Frameworks and Libraries these days, and rightly so, without them some projects I’ve worked on would still be under construction!!! Developing with frameworks and libraries will save you time in both development and testing.

The abundance of functionality some provide often mean you’ll end up with a better end product. Not all clients can afford the time and money required to have developers work from the ground up. With the benefits of useful frameworks/libraries your clients will get more than they wanted for less than you quoted (or you could keep the float).

I could go on for hours listing examples I’ve played with over the years, but some of my personal favourites are:

Zend Framework
http://framework.zend.com

The leading open-source PHP framework has a flexible architecture that lets you easily build modern web applications and web services.

Yahoo! User Interface Library (YUI)
http://developer.yahoo.com/yui/

a set of utilities and controls, written in JavaScript, for building richly interactive web applications using techniques such as DOM scripting, DHTML and AJAX. The YUI Library also includes several core CSS resources.

jQuery
http://jquery.com/

jQuery is a fast, concise, JavaScript Library that simplifies how you traverse HTML documents, handle events, perform animations, and add Ajax interactions to your web pages. jQuery is designed to change the way that you write JavaScript.

I realise there is a great deal more I could and should be listing here… and probably some much better?! Please leave a comment I’ve love to here your favorites… and I’ll try include them in future posts!