This blog post comes about a year late, but who cares. This particular project is very important to me, as it was really fun, and my first steps to python/django and sysadmin tasks like LDAP, SVN/Git and friends. The project was written along with my friend Cephalon (Γιάννης Σπανός) under the guidance of our professor Χρήστος Σωμαράς. It is still available at cronos.teilar.gr, but you won’t be able to sign up unless you are a student at TEI of Larissa.
The Problem
My school had an irrational number of websites, providing crucial announcements. It is very hard to follow all of them (the knowledge gets lost when it is spread around), not to mention the unavailability of some during most of the weekends 
Actually, that belongs to the past I hope, I haven’t noticed anything recently, but in the past the problem was really obvious. Every one or two weekends a power cut was hapenning, even in exams periods. The web services were too many though, some students were not even aware of some (for example the career.teilar.gr website has info about job positions, grad school programs etc). Another very important problem is that most of those websites require different accounts, and I couldn’t even imagine how many students didn’t remember their credentials, and did a password reset every new semester (remember, we are not talking only about CS students here, actually they are the minority, and they are not an exception to that unfortunatelly). Last but not least, none of the mentioned websites provided RSS feeds, making it even harder to track them (some of the major ones do now though).
Brainstorming
So, what would be really useful is a web application that will combine all those announcements, in a syndicator-like page. Plus, a number of other information (like grades, the semester’s declaration, their student mail account, even a list of teachers) could be also included to that app. And, most importantly, we need a unified RSS feed for all those announcements, since some (like me) prefer desktop applications to read their news. Well, we don’t need announcements of every school or every teacher though, the student should have the ability to select what they want to view. Since the uni websites don’t provide RSS feeds, we’ll have to parse the html pages instead, and store the output in a db. Some of them require authorization first, so we’d need the user’s credentials, which should be encrypted with blowfish in order for the system to be able to reuse them when needed. Everything looks good so far, let’s move on:
Implementation
First step was to set up the server. We got a new box and needed to do everything from scrach. We installed and configured OpenLDAP, MySQL, Django with mod_python, Subversion, phpLDAPadmin, phpMyAdmin, WebSVN, and the apache vhost files. Afterwards, we started messing with pycurl and BeautifulSoup. It took us some days, but we finally had two python scripts (django standalone files actually). The first was parsing the names of the schools, the teachers, the lessons as well as the names of the web services from which we were going to collect announcements. The second was actually parsing the announcements, of about 10 websites, which were put in a database. As I said there were no RSS feeds, so we had to send thousands http requests with pycurl and parse with beautifulsoup. I know, very unreliable, but there was no better way we could think of, plus it worked just fine. Some of the websites:
- www.teilar.gr: The most important website, it contains the announcements of all the teachers, school specific announcements, even college and various other announcements.
- e-class.teilar.gr: Equally important. e-class is a very popular webapp among greek universities, which offers tools for better communication between teachers and students. Apart from announcements, it also provides the ability to upload and store files, like presentation files or weekly projects, a calendar, a mini-chat, contact form and others. Many teachers prefer that to announce things instead of the first one.
- dionysos.teilar.gr: This one has the accounts of the students. This is where the student sees his grades, declares classes for the semester, etc. School wide announcements are also provided.
- myweb.teilar.gr: webmail service (no announcements provided here obviously )
- some informational only websites: LinuxTeam, PR, Library, Career, NOC and others (still adding new ones actually).
The above python scripts were put in cron.d, the first runs weekly and the second hourly. It takes about half an hour for the second to complete, and in the old box (a very old celeron laptop used as server) it took 45-50 mins. After having all that data in the db, it is now time to create the signup/login systems, and then print those data to the users. The login system took a while to get ready. We played a lot with LDAP, python-ldap and some django authentication backends to fully understand everything. We used the library ldap_groups, which pretty much had everything we wanted, although we had to tweak it a bit. In short, django allows the usage of another authorization backend instead of its ModelBackend, by adding the following in settings.py:
AUTHENTICATION_BACKENS = (
'path.to.custom.backend',
'django.contrib.auth.backends.ModelBackend',
)When a user logs in, the custom backend (assuming it is an LDAP backend) searches for the user in the django DB first, and if not found, it searches in the LDAP server afterwards. If the user is found in the LDAP server, its data will be transfered to the django DB. With that system, the data will stay in the LDAP server (and will be used easily) even if the DB gets wiped out. Similarly, the signup system was searching for duplicate entries only in the LDAP server. The signup got a bit more complex though. The system is asking for the accounts of dionysos, e-class (optional) and webmail (optional), and in order to verify them, it uses them immediatelly, performs a pycurl and parses the output. If successfull, it also parses the student’s data, like his name, semester, registration number, grades, e-class lessons he is subscribed, and put them in the ldap server. Using the django syndicator module we were able to create a unified RSS feed of all those announcements. The pages we created were the following:
- The first page which prints some personal information
- A syndicator-like page with all the announcements
- An e-class specific page, which shows the lessons a student follows, recently uploaded files, and pending deadlines for projects
- An e-mail page, which only shows the mails
- A library page, where the user can get results for books through the library.teilar.gr web page
- A list of all the teachers, with the school they belong and their emails
- An about page
- A settings page, where the student can change his password, his credentials of the other websites, update his grades/declaration/e-class lessons list, and select the teacher/other announcements he wants to follow
The future
Our target was to merge the LDAP server we created with the one the school uses. But due to some changes the NOC did, it was impossible to do that, so I had to drop the LDAP support in our application as well, and use the DB only.
After getting it online, two other guys created a similar django project as their thesis project, which would offer online registration to labs. The original plan was to merge those projects, due to many similarities in the code (especially regarding the registration system, which was also using the same LDAP authorization), but went on a separate web app finally, diogenis.teilar.gr (by Στέφανος Χρούσης and Γιώργος Τσιώκος).
About a month ago, the two services were moved to a new box, who was put under LinuxTeam control, and features two VMs (more on this on separate blog post). We have a long todo list now, and people that have interest in contributing. The service is still online, I hope it will stay for a while, thanks to Δημήτρης Παπαπούλιος and Άλεξ-Π. Νάτσιος for taking care of the server and the VMs, Γιώργος Κούτσικος for the artwork and Γιώργος Τσιώκος for the design! Let’s hope it will stay online for much longer, and more thesis projects on top of this will follow.
