Zope Page Templates in Google App Engine

written by Daniel Marks, on Feb 4, 2010 9:50:00 AM.

At EuroPython last year, Matt Hamilton gave a talk on using Zope Page Templates (ZPT) outside of Zope. ZPT uses the Template Attribute Language (TAL) to create dynamic templates that you can use in your own web applications, reporting frameworks, documentation systems or any other project.

Why TAL?

The point of this post isn't to go into detail about why you should use ZPT/TAL. Suffice to say that TAL:

Makes well-formed XHTML easy
Ensures that you close all elements and quote attributes
Escapes all ampersands by default & -> &

The Django Templating Language:

<ul>
  {% for name in row %} 
    <li>{{name}}</li>
  {% endfor %}
</ul>

TAL:

<ul>
  <li tal:repeat="name row"
      tal:content="name">
    Dummy data
  </li>
</ul>

Using ZPT in your own project

There are three steps to using ZPT in your own project.

Install ZPT (via the zope.pagetemplate package and its dependencies)
Create a template file
Render the template file using the data from your application

Install zope.pagetemplate

I recommend using virtualenv for each new application.

# virtualenv zptdemo
# cd zptdemo
# bin/easy_install zope.pagetemplate

Create a template

mytemplate.pt

<html> 
  <body> 
    <h1>Hello World</h1>
    <div tal:condition="python:foo == 'bar'">
      <ul>
        <li tal:repeat="item rows" tal:content="item" /> 
      </ul> 
    </div>
  </body>
</html>

Render the template

mycode.py

from zope.pagetemplate.pagetemplatefile \
    import PageTemplateFile

my_pt = PageTemplateFile('mytemplate.pt')
context = {'rows': ['apple', 'banana', 'carrot'], 
           'foo':'bar'}
print my_pt.pt_render(namespace=context)

And that's it. This will generate the following:

Hello World

apple
banana
carrot

Google App Engine

This is all very well if you're able to install your own packages, but Google App Engine (GAE) doesn't allow you to do this. You can, however, include packages with your application, which is what we'll do here. When installing zope.pagetemplate into your virtual environment, you may have noticed that the following packages were installed:

zope.pagetemplate
zope.i18nmessageid
zope.interface
zope.tal
zope.tales

We can extract all these files and put them into our GAE application.

I've already done this for you, and you can grab a copy of the 'zope' folder on GitHub here. Put this folder into the top level of your application.

Create a template as described above, and then place the Python code that renders the template into your application's RequestHandler. It should end up looking something like this:

class MainHandler(webapp.RequestHandler):

  def get(self):
    my_pt = PageTemplateFile('main.pt') 
    context = {'rows': ['apple', 'banana', 'carrot'], 
               'foo':'bar'}
    self.response.out.write(my_pt.pt_render(namespace=context))

At this point you have ZPT working in GAE and you can take advantage of all the features that ZPT and TAL provide.

The application on GitHub can be download, tested and even deployed straight to Google App Engine without modification. You can see an example of it running here.

update: it's also possible to use Chameleon on Google App Engine, which as Martin says may be easier and faster.

Plone at Europython

written by Matt Hamilton, on Jul 8, 2009 11:38:00 AM.

Last week Netsight were at the Europython 2009 conference in Birmingham, UK. The main reason we were there was to learn about all the cool stuff happening now in the Python world, but we also wanted to promote Plone as well.

The Plone Stand

To be honest it was a fairly last minute decision as we have just been so busy at Netsight in the past couple of months with client work. Also most of the Netsight team going (Matt, Daniel, Adam, Scott and honorary Netsight member, Dan F) are mainly Python and Zope developers, not really Plone developers. That said, as I've said before, nowadays you don't need to know that much Plone-specific knowledge to be useful in a Plone context, as much of it now is straight forward python development.

The Europython team graciously gave the Plone Foundation a space to have a stand at the conference free of charge. We just needed to bring the materials with us. I wasn't organised enough in time to get any of the bulk order of Plone stickers the foundation had printed up sent over, or get any more printed up locally. We were also traveling by train to Birmingham so didn't want to take too much stuff with us. We mainly took our large vertical Plone banner, a stack of Plone bags, some of the 'Top 15 Questions About Plone' leaflets and some of our Plone brochures with us.

Matt in front of the Plone stand

It was actually quite a different feel, as most of the marketing material we've been creating at Netsight for advocating Plone has been aimed at expos such as Internet World, and IMS in which we are competing against a lot of commercial players. As such much of our material was 'preaching to the converted' in terms of 'What is Open Source?' and the likes. For developers at a conference like this I think we really just needed lots of cool schwag to give away. Items like the Six Feet Up Plone bottle openers or t-shirts would have gone down well.

A few people came up to the stand to come talk to us to find out about Plone, and see what it is up to these days, though we didn't get as many as I was expecting. I think most people either 1) know about it 2) think they already know about it ;) or 3) don't care as they use X instead and why would they want to learn about other systems.

There were some other Plone development companies and individuals there, and we had a great Thai dinner one evening with Martijn and Roel from Four Digits and Laurence Rowe from Jarn. We also bumped into quite a few of the usual faces from the Plone and Zope world: Aiste Kesminaite (POV), Kit Blake (Infrae), Martijn Faassen (Startifact), Christian Theune (Gocept), Chris Withers (Simplistix), Christian Scholtz (COM.lounge), Charlie Clark (eGenix).

Next year I want to try an arrange a bigger Plone presence at Europython, as there were actually no talks on Plone at Europython at all this year (versus about 13 that mention Django in their abstract). I especially think a 'State of Plone' talk (like the ones done by Geir Baekholt and Alex Limi at the European and US Plone symposiums) would be a good idea. A tutorial on doing simple pythonic things in Plone such as creating a viewlet in Plone using Grok would be a great way to show the rest of the python community how easy it can be to get involved in Plone these days.

There have also been a great number of tools that have come out of the Zope world, and used extensively by Plone. The most immediately obvious one is buildout, which I think would be a great tool for with wider python community, but most other projects haven't come across it yet. Martijn Faassen did a great talk on 'Things that I helped Create' which was a journey through his life creating things starting as a kid. It was a fantastic insight into the creative process. Alas he ran out of time before he got onto the more recent items, in which he was going to be talking about some of these tools. One of the Four Digits guys was due to do a Lightning Talk too on Plone, but again time ran out before he had his chance.

-Matt

Photo credits: SafPlusPlus and MrTopf

Zope mindshare at Europython

written by Matt Hamilton, on Jul 2, 2009 9:31:00 PM.

I'm just on the way home now from Europython 2009 along with some of the rest of the Netsight team. The conference has been massively inspiring, with nearly 100 talks over the three main conference days. A massive thanks to the organising team, who did a great job of both the logistics and social side of the conference, and of course to all the speakers.

Conference Attendees

I did two talks this year, one a case study on a project we are currently working on using WSGI and Deliverance to skin a legacy .NET portal entitled Lipstick on a Pig. The second was an attempt to try and show how you can use some of the technology used by Zope outside of Zope: in this case a beginners' talk on Zope Page Templates. When I submitted the talk for the Zope page templates, the response from the talks team was 'Great! Finally a Zope talk!' as they hadn't had any yet. Ironically my talk was actually about using Zope stuff *outside* of Zope. I put a call out on twitter urging some more people to submit Zope/Plone talks, but alas it seems not many were forthcoming.

When I first attended Europython back in 2004, before any of the other frameworks existed, there was actually a dedicated Zope track at the conference, and there were a load of Zope and Plone talks there. It was actually a bit of an odd feeling, as you had a very distinct split in the conference attendees: those (mainly academics) that did hardcore stuff writing python compilers and simulated particle physics; and those people out in 'the commercial world' developing web apps with python in Zope and developing Plone, Silva, etc.

There was certainly a feeling that those doing Zope work were 'outside' the rest of the python community to a certain degree. This was mainly due to Zope being a trailblazer in terms of what it was doing and hence having to develop quite a lot of its own libraries and practises. Examples of this are libraries such as the DateTime library that Zope had before python had anything similar. I guess in just 'getting things done' some of Zope was maybe not quite as pure as python academics might have wanted, and Zope was a fairly monolithic system with little practical chance for its code to be used outside of Zope.

As a side note: in one of the keynotes this year Sir Tony Hoare talked about the differences between Scientists and Engineers. The former chasing absolute perfection, validation and proof in an ideal world; the latter concerned with an imperfect world and doing only exactly what is necessary to achieve the specification. This ties in with my feelings above, and it could be said that at that time the Zope people were the engineers and the rest of the python academic community the scientists. But times have moved on.

A year or so later the 'Zope track' became the 'web framework track' and Django, turbogears, pylons, etc joined in. This year the talks were completely mixed up together with commercial and scientific talks interspersed. This gave the event a much more coherent feel, and has to me been the best, most friendly, most inclusive Europython I've been to. Steve Holden, Chairman of the PSF, said that in his after dinner speech: Python really is about the people. Bruce Eckel had similar feelings in his keynote when he said after a stressful flight and journey to get here he walked into the conference and immediately relaxed with a sigh saying 'Ahhh... python people'.

I really agree with them and I think that python really is a very friendly environment to work in, both the language itself and the amazing community around it.

That said, we have a problem...

Looking at the talk abstracts for Europython there are 97 talks listed. How talks have the word Plone in the abstract? Zero. How about Grok? Zero. Repoze? Zero. Zope? One. That's my talk I did on using Zope Page Templates outside Zope. Silva? Two.

C'mon people, this is shocking! Zope and related projects and technologies have nearly completely dropped off the radar at this conference.

How many talk abstracts mention Django? Thirteen. Turbogears? Two. Pylons? Three.

Today Zope is a very different thing to what it was back then, with the entire Zope 2 application server being eggified and easy_install'able. The Zope Toolkit (previously known as Zope 3) also a collection of independently usable eggs. Technologies such as the ZODB, Page Templates, and Component Architecture are all usable outside of Zope and can be used in general python work. Projects such as Repoze are splitting things up further and allowing Zope to be used in a WSGI stack and re-using parts of the Zope Toolkit to produce repoze.bfg a lighter weight framework. We have zc.buildout which is an amazing tool for deployment of not just Zope projects, not just python projects, but pretty much anything. Grok, a layer on top of the Zope Toolkit provides a very rapid 'convention rather than configuration' approach to MVC web development, much like Rails does for Ruby.

But... I don't think the rest of the python community have quite got this yet. Maybe they still see Zope as 'that strange beast from years back', maybe the Zope community concentrates its resources on speaking at other events, e.g. Plone has not only its annual conference this year in Budapest (which has the same order of magnitude of attendees as Europython, but exclusively focussed on Plone) but additionally both a European Symposium and and US Symposium. That is a lot of time people will be spending traveling and attending and talking at events, but I think we really do need to get some more visible presence at wider python community events. We need to make sure the rest of the Python community see all the fantastic code and products that have come out of the Zope world.

There was a great talk by Martijn Faassen on 'Things I Helped Create' which was a breakneck speed journey through his experience in creativity in general from a small kid to where he is now. Unfortunately he ran out of time before he got really stuck in to all the Zope stuff he has done. It was still a massively enlightening talk. Christian Theune did a tutorial (alas I didn't make the tutorials) on using the ZODB for persisting objects, which would have also made a great talk (or at least lightning talk).

So this is a call to action. Next year Europython will be back again in the UK, and run to the same fantastic standard it was this year. And I want to make sure that there are more Zope/Plone/Grok/etc talks. Specifically I will be banging the drum come next year and really pushing people to do talks.

I'm even going to go out on a limb here and propose a starter list of talks:

Using the ZODB to Persist Objects
Using buildout to deploy stuff
The state of Plone
Introduction to Zope Component Architecture
Building a Grok app in 15 minutes.

If you want to find out more about what has been going on at the conference, Reinout van Rees has been doing an excellent job liveblogging the conference.

We need a Zope developer!

written by Matt Hamilton, on May 21, 2009 10:18:00 PM.

We have a requirement for an experienced Python/Zope developer to come work on a client project with us here in the UK. The contract will be for approx 10 weeks starting as soon as possible, and will be based in our offices here in Bristol with some time on client site. Unfortunately telecommuting in this instance is not possible -- we need someone physically on site here.

You will be working alongside a couple of other great guys, working on a system that combines Zope 2 and SQL (not Plone) adding additional functionality to an existing system. It is hoped that the new code will be written in a much more modern style using Zope 3 libraries and Five, but a good portion of it is likely to be old-skool Zope 2 development via the ZMI. The role will also involve some HTML/CSS work and writing ZPTs.

So, if you are interested and able to come over to the UK to start work very soon (we can arrange accommodation, etc) then drop me an email (matth@netsight.co.uk) with some details of yourself and your experience.

Plone in the Enterprise: Two Killer Features

written by Matt Hamilton, on May 16, 2009 2:27:00 PM.

Two weeks ago, we had a stand at Internet World in London. One thing that happened quite bit was people came up to us asking 'So I've heard of Drupal/Joomla, what does Plone offer?' and I would explain to them about some of the features Plone has making it ideal for deployment and integration in an Enterprise. I was going to write up about some of these points, but didn't get a chance to until now as something has pushed it back to the front of my mind again:

I've just been in a case study presentation of the Honda Italia Intranet at the European Plone Symposium in Sorrento. They have built a fantastic intranet in conjunction with Abstract Open Solutions. What was great to see was the process they went through in ending up using Plone. They worked with a consultancy firm who helped them to work out what their Intranet requirements were, and what they needed to be able to do.

They went through a very thorough process involving chatting with over 100 stakeholders in the organisation directly, and getting about 1000 staff to respond to a survey on what was going on. Card sorts were done, IA maps were drawn out, requirements prioritised. Only after all of this did they start looking at Content Management Systems and after doing a comparison of three Open Source and three Proprietary systems, they chose Plone.

I'm going to present just two specific killer features Plone has that sets it way above some other CMSs, both Open Source and Commercial.

Authentication

Something that Bruno Ripa from Abstract talked about in the presentation was Honda's complex authentication requirements. Well I say 'complex', but in fact from our experience its actually pretty normal within any large corporation. Their user authentication was spread over three active directory systems, one LDAP server, one SQL server, and a web services system. They needed to be able to combine all of them to provider all of their user information to their intranet -- a requirement that put quite a few, even large commercial, CMSs out of the running.

Plone is built upon Zope which allows us to us Zope's Pluggable Authentication Service (PAS). It is a powerful and flexible system that allows you to have separate (and multiple) plugins to handle all the different aspects of users. There are in total about a dozen different steps that plugins can be used for, but just a few of these steps are:

Extracting credentials from the user
Authenticating the user
Storing and managing properties and information about the user
Searching and listing users and groups
Assigning users to groups

This allows you to easily configure Plone to fetch information about a user from a multitude of sources around a corporation. You could authenticate them against several sources, and then fetch information about the user from a multitude of other places and combine them all.

Its flexible enough to allow you to use a variety of PAS plugins each to do the specific functions you want without having to use all or nothing. It also allows you to take the majority of your information from a central corporate source but then override or augment some of the information from a local source. Great for when you want your intranet to store some specific additional data on your users that is not suitable to go in a central source (or the powers that be won't let you!).

For more detailed, technical view of this and how we use it in a real example, see this blog entry on PAS.

Internationalisation

Another really killer feature for Plone that is lacking in quite a few CMSs is the ability to support and manage multilingual content.

By default Plone comes out of the box translated into over 50 different languages, and has a very simple system for allowing 3rd party developers and site customisers to provide translation strings for their products. An add-on for Plone called LinguaPlone provides a simple mechanism for creating and managing translations of pieces of content in your site. Once installed you simply choose 'Translate into...' and pick a language. Plone will then create a copy of your piece of content and present you the two versions side-by-side ready for translation.

The translate menu

Split Screen Translation

The contents language variants are linked together and even if you move a piece of content to somewhere else in the site it will remain linked to its translations. This allows you to have different site structures for different languages, but still easily switch between languages of a piece of content.

This product can simply be added into Plone at any stage of a site's life and viola! you have multilingual content support with a minimum of hassle. This means unlike with products like Sharepoint, you don't have to plan ahead to use multilingual features from the outset, and it allows you to translate an entire site, or just a few pages as necessary.

Pluggable Authentication Service

written by Matt Hamilton, on May 16, 2009 2:16:00 PM.

This is a more technical look at one of Plone's great features for the Enterprise, its authentication system. If you just want to know the high level benefits then read the previous post, if you want to know the nuts and bolts of a real world scenario then continue reading...

To recap, Pluggable Authentication Service (PAS) allows you to break up the authentication process into small steps and have individual plugins deal with each step.

But how does this work in reality? We have a client we are developing an intranet for whom need NTLM authentication. NTLM is quite a complex challenge/response mechanism, but allows users on Windows PCs to be transparently logged in to a web server using the credentials they supplied when they logged in to their workstation. We had to write a plugin for this as there wasn't one for Plone around (and we'd already done some NTLM work many years ago, so knew how it worked).

The NTLM plugin was responsible for telling the web browser to use NTLM (ChallengePlugin); extracting the credentials from the user (ExtractionPlugin); and actually authenticating the user (AuthenticationPlugin). So that gets the user authenticated, but all we know is the users username then (which in this company is just a random number). We need to get some more information about them such as their real name and their email address. Well there is an existing add-on PAS plugin that can get that for me, LDAPMultiPlugin. I can configure the LDAP plugin to look up information from Active Directory: the user's name and email address (PropertiesPlugin); Search/List all users in the company (UserEnumerationPlugin). I can also use the same plugin to lookup what groups a user is a member of (GroupsPlugin).

So, we are authenticating a user via one source and then looking up further information about the user and their groups from another source. However we want to assign the users roles in the intranet. This information is just specific to the intranet, and so whilst could be stored in LDAP it probably wouldn't make much sense, and we should probably store it locally. That's not a problem, as Plone comes with the ZODBRoleManager plugin which stores the roles locally.

We can even take this a step further: as setup above, every request to the web server causes a lookup to the domain controller server to do the NTLM authentication, which is a bit of a waste. Plone comes with a 'session' plugin which uses a cryptographically signed cookie to authenticate users which can be enabled for extracting credentials (ExtractionPlugin) and authenticating (AuthenticationPlugin). We now have two separate authentication plugins, NTLM and session. Plone will try both of them to authenticate a user and stop once it is successful, so if we put the session plugin at the top of the list that will be tried first, then if that doesn't work (ie this is the user's first visit to the site this day) it will try the NTLM one.

Technologies besides Plone

written by Matt Hamilton, on Feb 17, 2009 11:26:00 AM.

So thought I'd do a quick update of what we've been up to in the past couple of weeks, as things have been pretty busy here at Netsight.

First thing to mention is that whilst we are primarily known for our Plone work, the three largest projects that we are working on at the moment are not based on Plone. Aha! the doomsayers say, Plone is declining etc etc. But in fact the opposite, this is a positive thing... let me explain:

One of the main criticisms of Zope and Plone back when they started was that they were not very 'pythonic' ie. everything in Zope and Plone was done differently in every other python system. Some would say that Zope was so ahead of it's time that there wasn't anything else out there, so it was blazing its own trail.

Zope 2 was a very monolithic system and you basically had to use the entire stack. Not only that but every part of the stack was different to any other stack out there in the world. It was a big learning curve. The 'Z' shaped learning curve. Plone came along and hid a lot of this complexity to the average user, and simple customisation could be done fairly simply. As Plone got bigger, things got more complex, and some really clever stuff to make core developers lives easier made some simple customisations more tricky. This is now being addressed, and for Plone 4 much is being removed, creating a much sleeker system.

However the biggest wins for the likes of companies like Netsight that have to recruit, train and retain developers is that there is much more transferable code and skills between Zope and Plone and the rest of the python world. Since much of Plone nowadays is developed in the much more flexible Zope 3 style using Component Architecture you can transfer between other frameworks like Grok repoze.bfg, vudo... and not just the Zope based ones... but its now much easier to move back and forth with WSGI based frameworks like Werkzeug, Pylons, etc.

This is such a tangible benefit for a business. We have recently hired a new developer to work on one of the projects below, and we are currently looking for yet another developer to join us. What is great, is we can look for just 'python' developers, not specifically 'plone' developers as they don't need to know 'plone' now like they used to in order to be productive in a Plone environment.

To give some examples of this, let me talk about a few really interesting projects Netsight is working on at the moment... none of which directly involve Plone, but which all use skills transferable to and from modern Plone development. A no doubt Plone will be stirred into the mix at some point in both these projects' near future.

The first is a several month support and development contract with a large academic-related business. They employ 1,500 staff and their business is interacting with teachers and students across the globe. They built (using internal people) several Zope based web systems for managing the interactions with these teachers and students. These were built quite a few years ago, and are in need of a bit of a refresh and can do with moving to Zope 3. With the help of another company they have moved most of the code out of the ZODB and onto the filesystem so it can me managed much easier. Our first task is to build an events management system for them. They run events worldwide, and this system will take care of bookings and related tasks.

What makes this project so interesting is that we are able to now develop using Zope 3 style and technologies via Five on a Zope 2 existing system. This is really nice as it gives us a much nicer development model (classes, views, adapters etc) rather than the old style of a bunch of 'Script (Python)' objects. It makes it much easier for us to debug (via pdb) and to benchmark and test. Not only that, but we have a new developer working on the project, who whilst has python experience, has little specific Zope experience. Working in this way he has been able to concentrate on the pythonic aspects and leave the core Zope tasks to the more experienced developer on the project.

The second project is really quite a fun challenge. We are dynamically re-skinning an existing .NET portal site... without touching the original site. We are using a WSGI stack with modules that we are developing based upon xdv to dynamically clean up the (nasty, broken) HTML from the original portal and present a nice clean, reskinned site to the end user. Oh, and as we are gluttons for punishment, we are re-writing the URLs of the site too...

The Netsight Blog

written by Matt Hamilton, on Feb 10, 2009 6:03:00 PM.

OK, so I'm going to let you into a little secret here... this new blog, for one of the UK's longest established Plone development companies, isn't actually written in Plone. Shock! Horror! How can this be? Well our own website was actually designed and built before Plone even really existed. Before Plone, we did all our development in old-skool Zope 2. I guess it's a testament to our designers that even since the last time we re-designed our website nearly 7 years ago our website still gets a lot of praise for its design.

For several years now we've been saying 'we need to get a blog sorted out', but just never got around to it. We've developed countless blogs for clients, but never had time to do our own. In case you are wondering then why this new blog already has a history of articles, that is because for the past year we've had a mailbox setup at work that we've been emailing blog posts to ready for when we got the blog sorted... which of course was only meant to be temporary... (yeah, right!).

So what is this blog running on then, if it isn't Plone? Its running Zine a python based 'clone' of Wordpress. This was partly an experiment and no doubt will change a few times over the life of this blog. So why Zine then? Well, we have been starting to use a technology called Deliverance that allows you to apply a look and feel (theme) to a website without actually touching the site itself.

This was originally created for Plone to allow designers to create themes for Plone without having to know all the guts of Plone and get their hands dirty with python code and viewlets and the likes. It also allows you to apply the same theme to multiple sites and make them look coherent. The upcoming new plone.org website (which Netsight did the visual design for) is going to be using Deliverance to theme both Plone itself and also Trac, the issue tracking tool that the Plone project uses. In our case we use Deliverance to theme both Zine and our own Zope website.

In fact though, Deliverance doesn't theme our website, but uses our website as the theme. Yes that's right, you can specify a URL for the theme file for deliverance, so this could be a local file, or another site somewhere. So we had great fun playing around with this, theming our blog with various other public sites, Ikea, Disney etc etc. And the fantastic thing about it all was that we needed to touch neither the theme source (ie Disney's site) or the content source (Zine). This meant I could setup Zine nice and quickly without needing to get to grips with its (frankly, weird - a separate rant) way of handling theming and customisation.

For those interested in the technical details I'll be writing a subsequent post with technical details of the setup of our site and blog and how we are combining zope, zine, wsgi, deliverance (xdv) all together as one. I'm also hoping to write a bit about the cool project we are working on using the concepts in Deliverance to theme an existing large legacy portal for an organisation.

Sub-directories in a skins directory

written by Ben Ackland, on Jun 20, 2008 11:58:00 AM.

If you have sub-directories (e.g. "banners") within one of your registered skins directories, and you disable debug mode, you must have the "recursive" option set in skins.zcml for plone to traverse into the sub-directory, e.g.:

  <cmf:registerDirectory
     name="netsight_bestofbelron_custom_images"
     recursive="True"
     />

Setting the timezone in Zope

written by Matt Hamilton, on Jan 2, 2008 12:30:00 PM.

We have a Zope instance running on a central corporate server that needs to have the local timezone of another country (in this case the 'Belgium' intranet zope instance is hosted on the 'UK' zope server).

To do this add the timezone to the environment section in etc/zope.conf e.g.

<environment>
 TZ "Europe/Brussels"
</environment>

1, 2

Blog

Latest Posts

Archive