LotSo's OSS World

Postgresql 8.4 -> Where are On Disk Bitmap Indexes?

Sun, 05 Apr 2009 12:22:01 GMT

Postgresql 8.4 is nearly out. There's quite a few things which looks interesting to me. However, the one thing which I'm still missing and am not able to find the status of is where or what happened to the On-Disk-Bitmap-Indexes which was supposed to come out for the 8.4 release.

Anyone from the Postgreql SQL Team would be privy to that info? can't really seem to find it on google.

Thanks.

FC10 VPN Setup to Windows PPTP using Network Manager

Sun, 09 Nov 2008 15:16:34 GMT

I was going through this for a couple of hours as I was trying to configure FC10 (moved from gentoo [for now]) and NetworkManager to connect to my office's VPN server running on Windows(R) ISA server.

I tried a variety of methods and configurations but I keep getting errors of the sort


Dec 26 15:02:00 localhost pppd[5483]: Plugin /usr/lib/pppd/2.4.4/nm-pptp-pppd-plugin.so loaded.
Dec 26 15:02:00 localhost pppd[5483]: pppd 2.4.4 started by root, uid 0
Dec 26 15:02:00 localhost pptp[5484]: nm-pptp-service-5480 log[main:pptp.c:314]: The synchronous pptp option is NOT activated
Dec 26 15:02:03 localhost pptp[5493]: nm-pptp-service-5480 warn[ctrlp_disp:pptp_ctrl.c:956]: Non-zero Async Control Character Maps are not supported!
Dec 26 15:02:09 localhost pppd[5483]: MS-CHAP authentication failed: E=691 Authentication failure
Dec 26 15:02:09 localhost pppd[5483]: CHAP authentication failed
Dec 26 15:02:10 localhost pppd[5483]: Connection terminated.

Authentication errors, but the issue was WHY?

Ends up, and this may not apply to you, but it definitely applies to me.

My office is using Windows ISA server (200x version I would think)

So, in Network Manager, there are 3 option boxes

Username
Password
Domain

so, I happily added

Username : lotso
Password : lotso's password
domain : lotso's windows domain

ended up I get those errors above.!!

......
.....
2 hours later and MUCH googling
.....
....

I tried
Username : lotso's windows domain\lotso
password : lotso's password
domain :

and it WORKED!!

Ideosyncracies!!

on the other hand, this is _not_ a bug w/ NM or PPTP, seems like per digitalwound, his one works fine w/ those 3 options

Foss,My 2008 Pictures

Sat, 08 Nov 2008 18:31:49 GMT

nowadays, the blogs on OSS is really little for my part.

Here's some pics instead

http://flickr.com/photos/lotso

python-2.4 to python 2.5 upgrade hell

Tue, 12 Aug 2008 15:11:21 GMT

It's been a while since the last post.

Not much been happening except that I'm missing out on life due to work commitments which seriously sucks.

In any case, been battling with the python upgrade on my gentoo box at home.

I've been having undefined symbol issues with pygtk and pygobject all through last week and I just solved my issue like 10 min ago and thus I can go to sleep already.

Main issue is the borking when I was compiling a python app which needed gtk support. Only thing, it, it also needed "threads" support which, for some reason, is not default turned on in Python2.5 but is on python2.4!

Hence, I spent the last week pulling out hairs trying all different permutations of WHY did it USED to work and doesn't now.

it ended up as a simple USE flag which was not checked during the ebuild checks.

Why.. Oh.. Why...

Automatic Raid Array Rebuilding

Mon, 23 Jun 2008 01:32:15 GMT

Hi guys, long time no post. Last post was at March and it's now already June.

Been busy as usual, however, not been dabbling as much as I "should" as I've been busy with other NON-FOSS related stuffs. (psst: I'm now heavily into photography. Went to shoot some Japan GT queens!! Kawaaiii)

Anyway, since this is a (nearly) purely an FOSS based blog, I'm gonna talk about my automatic Raid Rebuilding script.

You see, what happens is this, my postgresql box, (celeron 2x500GB in Raid 1) has a tendency to keep dieing once in a while for X reasons. (I have till now, been unable to locate the reason why it's dieing so often) I've tried to the write-all, read-all using dd but thus far, has not seen errors being thrown out. So, it's been a manual instance of...

go to work. see the email : Your raid has Died!
log onto the box, do the rebuild.

After a while, this just becomes tiring and I decided to fsck it and make it automatic.

Here's the script

#!/bin/bash

FAIL_DRV=`mdadm --detail /dev/md0 | grep faulty | awk '{print $6}'`

if [ -n "$FAIL_DRV" ]
then
echo "Detected degraded array : $FAIL_DRV"
echo "Starting automated array rebuild process"
mdadm /dev/md0 --fail $FAIL_DRV --remove $FAIL_DRV --add $FAIL_DRV
else
echo "Nothing to do"
fi

Simple eh..

So, now I don't have to come to work to see it all wonky because it'll automatically rebuild itself.

Some of you may ask, how come I don't just replace the drive? Because I can't find any replacement drive which is a PATA connection and at 500GB capacity! The largest I can find are 160GB.

Bummer

GVFS makes me happy

Fri, 04 Apr 2008 21:00:51 GMT

Did you know that nautilus is now integrated with the new GVFS (~~gentoo~~Gnome virtual filesystem) from the older gnomeVFS module?

The new one is partially built on top of fuse, or rather integrates with fuse and it makes mounting and accessing files from network shares a much better experience than it was previously.

GVFS, with nautilus, when you browse to a share, it’s automatically mounted under ~/.gvfs


gvfs-fuse-daemon on /home/gentoo/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev,user=gentoo)

~/.gvfs $ ls -al
total 12
dr-x------   3 gentoo users     0 Apr  5 03:38 .
drwx------ 118 gentoo users 12288 Apr  5 04:37 ..
drwx------   1 gentoo users     0 Feb  1 21:20 mediacenter on 192.168.10.111

and thus, you can actually use applications to access those items under those mount points.

for example, one of the reasons I rely heavily on totem and nautilus is because of its SMB support. Whereby I can connect to my home mediacenter server and stream videos from it without going through the motions of actually mounting the drive/shares.

(did you know that only totem and nautilus share this feature in gnome and the rest of the programs are brain-dead in this regard?)

Kinks :
Currently, based on my limited testing (it’s 4+am as I write this as I was playing with GVFS+Nautilus 2.22), there are bugs when you try to access a smb share which is password protected. For X reasons, the usual nautilus “password verification” box does not come up.

And putting smb://user:password@server/share does not work either.

But if you were to drop down to the CLI and do a


$ gvfs-mount smb://gentoo@192.168.10.2/storagePassword required for share storage on 192.168.10.2
Domain [HOME.NET]:
Password:

then that would work and you’ll get the mountpoint in ~/.gvfs and you can access files from that location.
It’s an additional step, but hey, at least now it’s

1. transparent (to an extent) and most apps can see it (tried mplayer/gmplayer/xine)
2. FAST. It’s pretty much faster than the previous incarnation of using smb protocol through nautilus. (I don’t have any real stats)
3. Have a tendency to crash.

Steps to secure your site

Thu, 06 Mar 2008 00:36:27 GMT

So, in today’s lesson I will elaborate on how 1 site decides to put additional “protection” towards phishing or in a more general term, how to secure your site against malware or other badwares.

1. Open an account with RHBBank (rhbbank.com.my)
2. Subscribe to internet banking
3. Go Overseas
4. Attempt to pay your credit card fees etc via internet
5. Pull hairs in attempts

So basically, I’ve been trying to access to RHBbank’s secure site (https://logon.rhbbank.com.my/) and keep getting either permission denied or server errors or something along those lines.

So, in an off-hunch, I tunnelled to my home squid proxy server and used that as the proxy for firefox. I fired up the browser and was greeted with the RHB secure page!!

Open up opera, (normal settings) and fire up the same page and “internal server error”

So, either one of two things is happening.

1. RHB is looking at IP addresses and denying access to anyone out of M’sia IP address range
2. My Company’s outgoing filter regards RHBbank as malware etc and prohibits me to visit it.

funny business.

Open Source Trend - M'sia 2nd in line

Thu, 28 Feb 2008 06:31:26 GMT

I just noticed this from google trends.

http://www.google.com/trends?q=open+source&ctab=0&geo=all&date=all&sort=0

Location..Location..Location

Tue, 05 Feb 2008 01:48:54 GMT

I’m in San Jose. Still pondering if I can make it to the Local PUG (postgresql user group) meeting to be held on Feb 12 since I’m here.

Will get the chance to meet David Fetter and team.

I’ll see what happens.

PS : I freaking hate it here this time of year. It’s cold and so are my fingers! I need to constantly rub my hands together

You say Lemon, I say Lemonade (A story)

Sun, 27 Jan 2008 09:51:27 GMT

The past few weeks was not all that great as in addition to facing additional challenges at my primary day job, I also had to deal with my pet project in my day job to help smoothen my day job’s activities.

Some of you may know that my pet project involves pulling gobs of data into a PG instance to make my own version of a company datamart. I’m not talking about small gobs of data, but more towards in the range of 200+GB (It was more, but in one of the efforts to control/tune the server, I deleted close to 2-3 month’s worth of data.)

200+GB may not seem like much to you guys who gets to play with some real iron hardware or some “real” server hardware. All I had was just a Celeron 1.7G w/ 768MB of ram and some Gobs of IDE 7200 RPM drives. In short, all I had was lemons and I needed to make the best of it!

Actually, all was working fine and dandy up until I decided to make a slave server using Slony-I + PGpool and while that was a good decision, the involved hardware was the same if not worst(512MB ram only). When I started to implement that, I was faced with 2 issues.

1. Replication would lag behind by up to a day or so waiting for the next sync (dreaded fetch 100 from log) was taking to long.
2. My nightly vacuum job went from an average of 4+ hours to like 27+ hours.

So, in a effort to get things under control, I went through a few paths and hit more than my share of stumbling blocks. One of the things which I tried was to reduce the amount of “current” data in a particular table from 1 month -> 2 weeks -> 1 week (and move them into a so-called archive table but still in the same tablespace). This didn’t really bode well, as I initially tried to move the data in like 3 hourly chunks, which failed and to 1 hour chunks and then finally to 15 minutes chunks.

But in the end, it was all really futile because what i was essentially doing was just generate more and more IO activity (and that’s not a good thing). In addition to that, I also had to deal with vacuuming the tables due to PG’s MVCC feature and that was also not fun.

So, in the end, I broke my 3x500GB Raid 1 mirror (1 spare disk) and used the spare as the Slony-I log partition. Initially, that wasn’t all I did, I also included the 2 main problematic table, moving it from the main raid1 tablespace into that 1 disk tablespace. (that was also a mistake) and it didn’t help at all. IO activity was still high and I wasn’t able to solve my vacuuming process as wel.

Time for another plan.

This time around, what i did was to move the 2 big tables back into the raid1 tablespace and left the slony logs in the single disk. In addition to that, I also made a few alterations to the manner in which I pull data from the main MSSQL database and the way it was inserted into PG.

This time around, I’m utilising partitioning and some additional pgagent rules to automatically switch into a new table every 7 days and in doing so, I also had to change a few more other items to get things to work smoothly. I did this last Friday and based on the emailed logs, I think I’ve made a good decision as right now, everything seems peachy with the vacuum back to ~4 hours and there’s also no lag in the Slony replication.

I still hav another thing to do which is to alter the script I use to pull from the main Db as I’m being kicked (requested) to pull from an alternate DB which has a slightly different architecture.

2 disk Raid1 is definitely MUCH better than a single disk tablespace. With the amount of read/write activity that i have, it’s just not doable.

So, that’s how I made lemonade with my lemons. (hmm.. does this sound right?)

Postgresql 8.3 Features I'm looking forward to

Sat, 12 Jan 2008 12:39:40 GMT

PG 8.3 is coming along soon. (although I read from Bruce M that there's likely to be RC2 coming out).

In any case, I looked through the pgwiki and there looks like only 2 features which I'm looking forward to.

HOT
Create table like including indexes (although right now, this is being automated via a stored procedure/function)

The other thing which is nice, but not absolutely necessary is the multiple Autovacuum worker feature. My concern is largely on the few very large tables which I used to have. (I've since sliced it down to partitions by date ranges to keep it manageable. I initially just wanted to see how _much_ data it can cope with before my system** starts to bog down. BTW, It turned out to be approx 200 million rows, and Now I know)

Of late, the nightly vacuum has been taking a long time and this is in part, a fault of mine due to a design issue. I won't go so much into this, but know that I need to relook into my current ETL implementation and where the data goes into the Db.

As of right now, I'm pulling data from a MSSQL server into PG to be made as a data-mart. My current process involves pulling from MSSQL into a table in PG. Unlike the usual method of making a partition, namely a master table w/o holding any data or insert directly into the partition, I chose to insert into the master table, and then, 1 week later (I started with 1 month then 2 weeks and ended up with 1 week's worth of current data in the master table) I start to offload data from the master table into the partition.

Master Table (1 wk data)
->partition_200710
->partition_200711
->partition_200712

I was looking through my system's load and found that it's always on IO wait. Performing a vacuum on the large table after the data offload into the partition took quite a while due to

The table is large
The indexes are sometimes even larger than the table size
The number of indexes in that table
My usage of a concatenated prikey named as unique_id to simplify the loading process which ended up being a bad decision because I needed to create the same prikey (non-concatenated as an index) anyway to improve join performance. Hence, in some sense, i have double the amount to vacuum through. Bad. Bad. (David Fetter warned me of this but I chose to shoot myself in the foot anyway.)

So, I figured that by reducing the amount of data in that particular table, I could well reduce the amount of time being spent in vacuuming that particular table. (Note that I don't know how true is this hypothesis of mine, but I'm giving it a shot anyhow.)

Note: I'm looking forward to 8.4, which I don't really know when, but I'm hoping that by then, (on disk) bitmap indexes will be made available and my (multiple) indexes can be made to be smaller and more efficient. (up to 8 index on a table)

** : The system in question is a celeron 1.7G/768MB RAM and 2x500GB Raid 1 w/ ~250GB DB size

The Doraemons Conversations

Thu, 10 Jan 2008 09:11:36 GMT

After waiting for such a long time and after the long wait to compile QT and also skype 2.0 (beta download) I finally gotten the el-cheapo webcam which I’ve gotten from nearly a year ago to work. (actually, I think it was longer than that, it was during version 1.3 IIRC of skype)

The driver used was the gscpav1 and for the webcam to work, you have to enable video4linux support in the kernel (which I didn’t bother to since skype doesn’t support video in the olden days anyway.)

So, after all that, I now have Skype for Linux working. (didn’t test sound though)

But I’m happy that I don’t need to spend RM65 to get another webcam like what colin did.

SQL - pgpool-II (Step 2)

Sun, 06 Jan 2008 06:24:57 GMT

So, this is step 2 to getting replication + load balancing to work for postgresql.

I've already detailed the 1st step to getting Slony to work in a previous blog. (that was on a development machine/vmware image. When I tried it on the production/slave server, I was faced with some issues which I might elaborate in another post. It all boils down myself shooting my own foot. What to do, it was a time when I wasn't connected to the internet and thus, no googling privileges.)

So, here are my experience with pgpool and it's also a little bit like shooting myself in the foot (again!)

First off, I started out with using the _wrong_ version of pgpool. The newest version of pgpool-II (note that -> pgpool-II and not pgpool-I) is 2.0.1 and the newest version of pgpool found on the yum mirrors (I'm using centos4/5) was 2.01 (well, the numbers match don't they?) The only different was the one available on the yum mirrors was that of pgpool-I and not pgpool-II. However, since documentation on pgpool were sparse (I googled everywhere, read all the relevant and NON-relevant mailing list and found nothing much to go on by.)

It was not until I signed up to the pgpool mailing list (which was very low volume by the way) and interacting with one of the Japanese developer did I find out that I was in-fact using the OLD version of pgpool which was pgpool-I which, unfortunately had the same version as pgpool-II!
(I even downloaded the tarball from pgfoundry[but I _did_ download the _correct_ tarball] and searched through the source to figure out what was happening.)

By that little(big!) mistake I did, I was tearing my hair out for the past 3+ weeks. (well, I didn't play with it everyday and in-between my dayjob and such....) However, I did get pgpool-I to work properly with a little tweaking and I could get load-balancing to work, albeit it was not as advertise as in I can't get it to work without it functioning as replication as well. (of sorts anyway, which was the reason I can't deploy it as I was using slony)

So, after I found out my mistake last friday, I started to google for a new RPM of pgpool-II (newest version 2.0.1) but was unable to locate it in any place. The latest RPM I could find was that of version 1.3 which was _too_ old in a sense. (It's always better to have the latest stable version) So, I had to engineer a way to get a RPM from the tarball. Luckily, the tarball from pgfoundry also contained the pgpool.spec file, which was packaged by Devrim. Unfortunately for my, the spec file was a little old in that it refered to the 2.0 beta1 version. It wasn't too much of an issue as all it needed was a little hack here and a little hack there. (I was getting bad owner/group permission error which I narrowed down to the .spec file not having valid user/groups.)

After that was done a rpmbuild -ba pgpool.spec and I got an RPM.

After that, I just installed it, configured the pgpool.conf and got it up and running as advertised with replication mode off, master slave mode on and load balancing mode on.

Cool.. I'm rolling this to production on Monday.

So, this means I'll have 1x Master (1.7G celeron/768MB ram, 500G Raid1 with ~200GB of data), 1xSlave (1.7G celeron/512MB ram 3x160GB raid0). I still have another box sitting under my desk which has even poorer specs than the above, but I think it'll work out just fine.

Cool..Ultra Cool Even!!

If anyone wants the RPM or the modified spec file, do drop me a line and I'll post it to you or something.

SQL - Slony-I (step 1)

Sun, 23 Dec 2007 17:18:23 GMT

Been playing around with some level of replication for Postgresql. Like in all FOSS based software, there is lots of choices to choose from and that, in itself, though a blessing is also a curse. There’s just too many choices! (Both Foss and Non-Foss per se)

1. Sequoia
2. PgCluster
3. CyberCluster
4. Slony-I
5. PgPool
6. Skytools (this is skype)

and i believe the list goes on. In any case, my requirements are just 2 I think. (for now anyway)

i. Replicate only a subset of the tables. (not the entire db)
(AFAIK, pgcluster, while easier to configure is also an entire DB replication solution, which is not what I wanted)

ii. Connection load balancing to a few read-only slaves (for select queries only)

Hence, based on the overflowing amount of information of which option to choose, I finally arrived at using slony-I and pgpool and of the two options, I’ve (more or less) already completed the configuration of Slony-I.

For Slony-I, I made sure that I understood how to do the “old-style” which is by using the cli, before I moved on to doing the rest of the configuration using pgadmin which is way easier.

Slony-I
There are a few caveats when using Slony-I and I’ll list down my experiences when I’m playing with it using both gentoo and centos 4 (this is running in a VM)

1st off, version 1.2.12 is out from the slony-website but gentoo is still at 1.2.10. The easiest thing to do with this is just to hack the ebuild and change the version from 1.2.10--> 1.2.12 (gentoo bug #143600) and move it to /usr/local/portage.

So, in that sense, building on gentoo was relatively straightforward and less than 10 min job (excluding compilation)

But on centos, it’s another matters since there’s no default rpm supplied. Only a src rpm was supplied and not being too utterly familiar with it, (i’ve switched to using gentoo nearly 4/5 years ago as I hated fedora’s upgrade cycle and centos was “supposed” to be server-grade.)

In anycase, most of the caveats are when dealing with centos. For one, since this is a src.rpm, you have to compile it 1st.

Hence, you need these additional packages :

1. bison
2. flex
3. gcc (and all it’s dependencies)
4. rpm-build
5. postgresql-devel
6. docbook-style-dsssl
7. netpbm-progs (and netpbm dependency)
6. (there might be more as I didn’t document it)

Once you start compiling it, you’ll run into 1 error which is caused by the NAMELEN of the docs. (this is marked as bug #159382 and the solution is to either upgrade to centos 5 (supposed to be fixed by this release. Keyword = supposed) or to hack it. (I chose to hack it)

depending on where your docbook files are, you can do this

cd /usr/share/sgml && perl -pi.bak -e ‘s/(NAMELEN\s+)44/${1}256/’ ‘find . -type f |xargs grep ’NAMELEN.*44’|sed -e ‘s/:.*//’‘

So, after that is resolved, (which took 1-2 hours w/ scouring net etc.) Then move on to the experimenting stage. I used articles from these few locations :

slony-i official docs
WhoAmI’s Blog
OnLamp Article from 2005
Pgadmin Archives
Pgadmin Docs

Anyway, a few more caveats with the configuration is.

1. Ensure you use a .pgpass file for the passwords (chmod go-rwx ~/.pgpass)

192.168.10.100:5432:*:postgres:pguserpassword
192.168.10.20:5432:*:postgres:pguserpassword

2. Ensure that you use sane configs for your pg_hba.conf file (use trust/ident authentication 1st just in case, to ensure it’s not due to that if it’s not working)

3. ensure that the connection string used for slon/slonik also uses the “user=postgres” line.
(notice that this guide doesn’t have the user to connect as in the slonik shell script. This caused me some headache as I was getting both a password error as well as some “cannot connect admin node xxx issues)

4. Create the replication using either directly using shellscripts or using pgadmin3. (i followed both the examples from the pgadmin docs as well as the mail I found on the pgadmin mailing list - links provided above, with the exception that I didn’t make it 2 way as in slave<-->master but only master-->slave and slave-->master.)

5. starting the slon process is as simple as (I used a config file instead)
$cat > slon_master.conf
cluster_name = ‘pgcluster’
conn_info = ‘dbname=testcluster host=192.168.10.20 user=postgres’
^C

$slon -d4 -f slon_master.conf

(-d4 to give lots of debug output)

on the Master DB

2007-12-24 01:04:12 MYT DEBUG2 syncThread: new sl_action_seq 1 - SYNC 217
2007-12-24 01:04:16 MYT DEBUG2 localListenThread: Received event 10,217 SYNC
2007-12-24 01:04:17 MYT DEBUG2 remoteListenThread_1: queue event 1,195 SYNC
2007-12-24 01:04:17 MYT DEBUG2 remoteListenThread_1: UNLISTEN
2007-12-24 01:04:22 MYT DEBUG2 syncThread: new sl_action_seq 1 - SYNC 218

on the Slave DB

2007-12-23 22:05:02 MYT DEBUG2 remoteWorkerThread_10: SYNC 227 processing
2007-12-23 22:05:02 MYT DEBUG2 remoteWorkerThread_10: no sets need syncing for this event
2007-12-23 22:05:04 MYT DEBUG2 remoteListenThread_10: queue event 10,228 SYNC
2007-12-23 22:05:04 MYT DEBUG2 remoteWorkerThread_10: Received event 10,228 SYNC
2007-12-23 22:05:04 MYT DEBUG3 calc sync size - last time: 1 last length: 2005 ideal: 29 proposed size: 3

6. BTW, there’s no such need to do a database dump and restore of the tables you want to be replicated. It’s as good to just create the schema w/o any data and start the slon processes. I learned that all my effort to dump and restore the replicated tables just ended up in the drain as slony-I will just truncate the table (this was a command I caught a glimpse of when slon started) and restart from scratch. (i really wonder if this is intended behaviour. What happens when the slon processes goes down? and it seems that it’s quite fragile, so I’ll have to look into that.)

Next up is to look at pg-pool. That’ll be another fun(?) thing to look at??

BTW, I’m looking to do the replication to another (low end celeron) box and perhaps just do a raid0 out of 3 drives for greater performance(?) and then pg-pool to load balance it to the raid0 box.

Build performance and redundancy through multiple un-reliablie boxes eh? The google philosophy.
I’ve got a few low end boxes lying around in the office which can be put to use I suspect.

Gnome-2.20 - Totem Backend Changed to Gstreamer (By Default)

Tue, 18 Dec 2007 19:29:23 GMT

This sucks.. it’s 3am and I’m battling with Gnome-2.20 and the new Totem gstreamer backend which is refusing to play nice with RMVB files. (actually, I think it is unable to handle any codecs which is not supported by gstreamer - which is also why there’s the pitfdll plugin - which is not in gentoo’s portage by the way)

I really like totem as it integrates nicely with gnome (in general) and not to mention that it also is able to play/stream from a smb share unlike the rest of gnome and linux (in general and in my experience anyway). All other options, one has to mount the smb share into linux, or copy the entire file into the system before one can really play it, which totally sucks by the way.

It also seems that I was stucked using totem-2.16(I can’t remember why) and in totem-2.18, the gentoo people made the default totem backend from xine to gstreamer. (even though xine is still a supported backend based on what I read on the totem website)

So, going through the internet for “possible” solutions, I finally ended up hacking the ebuild to suit _my_ needs.


 $ diff -Nau /usr/portage/media-video/totem/totem-2.20.1.ebuild /usr/local/portage/media-video/totem/totem-2.20.1.ebuild
--- /usr/portage/media-video/totem/totem-2.20.1.ebuild  2007-11-29 14:06:23.000000000 +0800
+++ /usr/local/portage/media-video/totem/totem-2.20.1.ebuild    2007-12-19 02:55:44.000000000 +0800
@@ -103,7 +103,7 @@
        # use global mozilla plugin dir
        G2CONF=“${G2CONF} MOZILLA_PLUGINDIR=/usr/$(get_libdir)/nsbrowser/plugins”

-       G2CONF=“${G2CONF} --disable-vala --disable-vanity --enable-gstreamer --with-dbus”
+       G2CONF=“${G2CONF} --disable-vala --disable-vanity --enable-xine --disable-gstreamer --with-dbus”

        if use gnome ; then
            G2CONF=“${G2CONF} --disable-gtk --enable-nautilus”

Note : This at least made it able to play rmvb files once again.

Note 2: You may need to mae a symlink to your win32codecs install location as xine defaults to searching in /usr/lib/codecs (which doesn’t exists in gentoo)

SQL - pgadmin can't do table inheritance

Sun, 18 Nov 2007 13:10:27 GMT

So I found out that in some sense, pgadmin is like shooting oneself in the foot. (if you’re not in the know)

Let’s see how many times I’ve shot myself in the foot.

1. There was no option for moving indexes to a separate tablespace from within pgadmin (1.8)
--> This can be done using psql -> alter index xxx set tablespace fastspace

2. There’s no option for making a table to become an inherited table AFTER table creation. (Note that there is an option in the GUI for adding/removing inherit tables, but in mine, it’s greyed out.)
--> Via psql -> alter table footable inherit footable_parent

hmm.. only 2 times.. (at least that’s how many I can remember right now)

SQL - Perl DBI - Updating the rows counts

Sat, 17 Nov 2007 05:09:14 GMT

I’m syncing my PG database with the main MSSQL DB at a specified interval and I was wondering how many records were being deleted/inserted at every hour so that i can get a feel of how much latency there is between the main SQL server and my data mart.

My solution to pull data from mssql and insert into PG is based on Perl-DBI. Initially, I was wondering how I can get the rowcount (rows affected by a query) to be inserted into a log table. AFAIK, there isn’t a @@rowcount (mssql feature) in PG but there is a GET DIAGNOSTICS = ROW COUNT feature using pl/pgsql (which I use by the way when I write pl/pgsql) but since this was using perl-DBI, this feature wasn’t available. Hence the problem.

I researched it a bit and found out that Perl-DBI provides the metadata of the number of rows affected by NON-SELECT statements.

Hence, I set out to incorporate that into my script. (turns out that it wasn’t all that difficult)


  # We have a basic 4 SQL statement to execute.
  # DELETE / INSERT / UPDATE / TRUNCATE
  my $query0 = “TRUNCATE TABLE $table_name_loading”;
  my $query1 = “DELETE FROM $table_name
                WHERE $unique_id in
                (SELECT $unique_id from $table_name_loading)”;
  my $query2 = “INSERT INTO $table_name SELECT * FROM $table_name_loading”;
  my $query3 = “UPDATE log_sync SET last_sync=?,
                record_update_date_time=current_timestamp
                WHERE table_name=?
                AND db_name = ?”;
  my $query4 = “INSERT INTO log_update(job_name, table_name, from_date, to_date, rows_deleted, rows_inserted)
                VALUES (‘mssql_2_pg’,?, ?, ?, ?, ?)”;
  my $query5 = “TRUNCATE TABLE $table_name_loading”;
#DBI->trace(1);

  # The queries/SQL are wrapped into an EVAL because we
  # expect that these queries MAY fail due to duplicate Primary Keys
  # Note that these are all running as 1 transaction. If anyone failed,
  # we will call the errorhandler, rollback the changes, send email
  # and quit
  eval {
    print “Executing CLEANUP\n” if ($verbose);
    $sth_pg = $dbh_pg->prepare($query0) or die “prepare failed $DBI::errstr”;
    $sth_pg->execute();

    print “Executing DELETE\n” if ($verbose);
    #$sth_pg = $dbh_pg->prepare($query1) or die “prepare failed $DBI::errstr”;
    #$sth_pg->execute();
    my $del_rows = $dbh_pg->do($query1) or die “prepare failed $DBI::errstr”;
   if ($del_rows != 0)
   {
     print “Number of rows deleted: ” . $del_rows . “\n”;
   } else {
     $del_rows = 0;
   }

    print “Executing INSERT\n” if ($verbose);
    #$sth_pg = $dbh_pg->prepare($query2) or die “prepare failed $DBI::errstr”;
    #$sth_pg->execute();
    my $ins_rows = $dbh_pg->do($query2) or die “prepare failed $DBI::errstr”;
    if ($ins_rows != 0)
    {
      print “Number of rows inserted: ” . $ins_rows . “\n”;
    } else {
      $ins_rows = 0;
    }

    print “Executing UPDATE\n” if($verbose);
    $sth_pg = $dbh_pg->prepare($query3) or die “prepare failed $DBI::errstr”;
    $sth_pg->execute($to_datetime, $table_name, $mssql_default_db);

    print “Executing INSERT INTO LOG\n” if($verbose);
    $sth_pg = $dbh_pg->prepare($query4) or die “prepare failed $DBI::errstr”;
    $sth_pg->execute($table_name, $from_datetime, $to_datetime, $del_rows, $ins_rows);

    print “Executing TRUNCATE\n” if ($verbose);
    $sth_pg = $dbh_pg->prepare($query5) or die “prepare failed $DBI::errstr”;
    $sth_pg->execute();
  };

  errorhandler(“dbh_pg”);
#  $dbh_pg->rollback;
# If we got this far, then we commit the transaction
$dbh_pg->commit;
}

A portion of the code as highlighted above needed to be done because in the log_update table, I’ve defined the rows_inserted/deleted as integers and Perl-DBI, in it’s wisdom, output 0E0 when there is 0 (zero) rows affected by the query, for whatever reason. Hence those lines were added to ensure that 0 is outputed instead of 0E0.

SQL - Coolness Factor X

Sat, 03 Nov 2007 08:47:53 GMT

This is really cool and I can already see some ~~nasty~~ nifty stuffs to do with this nugget.

My objective is to see if I can do some kind of DB link from within postgresql to a Miscrosoft(tm) SQL Server instance.

There are a few method of implementation, namely

1. dbi-link
2. dblink
3. dblink-tds

In this blog, I will talk about using option 3 which is basically a method for getting access to a MSSQL or a Sybase DB instance from within PG.

The install process is quite simple (at least on gentoo, but I’ve yet to determine how to go about installing it on Centos, which is the deployment/target server)

make
make install

       query
        ------------
        dblink_tds(text, text, text, text) RETURNS setof record
                - returns a set of results from remote query (can be any kind of SQL query);
                - arguments are:
                - 1 text: SQL command string
                - 2 text: Name of the server entry in freetds.conf
                - 3 text: Username used to connect to MS SQL
                - 4 text: Password used to connect to MS SQL

        dblink_tds(text, text, text, text, int) RETURNS setof record
                - returns a set of results from remote query (can be any kind of SQL query);
                - arguments are:
                - 1 text: SQL command string
                - 2 text: Name of the server entry in freetds.conf
                - 3 text: Username used to connect to MS SQL
                - 4 text: Password used to connect to MS SQL
                - 5 int: Port number used to connect to MS SQL (default is 1433)

        dblink_tds(text, text, text, text, int, text) RETURNS setof record
                - returns a set of results from remote query (can be any kind of SQL query);
                - arguments are:
                - 1 text: SQL command string
                - 2 text: Name of the server entry in freetds.conf
                - 3 text: Username used to connect to MS SQL
                - 4 text: Password used to connect to MS SQL
                - 5 int: Port number used to connect to MS SQL
                - 6 text: Complete path to freetds.conf file (default is /etc/freetds/freetds.co
nf)

The only drawback from doing a source install and not from an ebuild is that the default install location is in /usr/local/pgsql but my postgresql library location is in /usr/lib/pgsql. Thus, i made a symbolic link as a hack.

Anyway.. post installation, I did a couple of testing and I’m happy with the results.

One example usage is

SELECT 
a.id, a.famid, a.dcm,b.supplier 
FROM d_part a
INNER JOIN (
SELECT * 
FROM dblink_tds($$select famid, dcm, supplier_name 
    FROM database.dbo.supplier_lookup b$$,$$NeuroXP$$,$$sa$$,$$11111$$) AS 
(famid text,dcm text, supplier_name text)
) b
on a.famid = b.famid
and a.dcm = b.dcm
WHERE batchid = ‘2002’;

take note that you have to put the AS (..) condition else it’ll spew out errors relating to wrong record type.

my freetds version is 0.64 and this is what I’ve placed inside as definition to the connection

[NeuroXP]
        host = 172.16.124.128
        port = 1433
        tds version = 8.0

Thanks to davidfetter in #postgresql again

SQL - Error!! Take Evasive Action (and not the other way around)

Sat, 27 Oct 2007 06:26:54 GMT

As I look more into the previous d_refresh(tblname text) function, the more I see that the checks for duplicates are taking up a whole chunk of server/cpu/disk time. (most important is disk IO time, even though the time it spends there are just seconds, it is seconds too long to check for duplicates IF there aren’t any to begin with!)

So, I embarked to make the function better.

In this case, I chose to NOT follow the adage, fix it before it breaks. This time around, I chose to “Fix it IF and ONLY if it breaks”. This will be a needed relief for the server.

So, I broke down the previous function and included an EXCEPTION check for unique_contraints. (I was looking through the Postgres Docs looking for error codes for duplicate primary key issues but can’t find any; at that time, I didn’t know that unique_contrainst == duplicate primary key until I saw this from Varlena.com)

Hence, curently the function is broken down to 2 different segments.

1. Normal Insertion, (delete/insert)
2. Delete duplicate Primary key. (this is different from #1 in ways which I can’t explain properly in writing w/o much effort. So, just trust me, its different.)

Function 1.


CREATE OR REPLACE FUNCTION d_refresh(tblname text)
  RETURNS void AS
$BODY$

DECLARE
last_r timestamp;
r_interval interval;
del_qry text;
ins_qry text;
del_stime timestamp;
del_etime timestamp;
ins_stime timestamp;
ins_etime timestamp;
max_time timestamp;
del_rows integer; 
ins_rows integer; 

tblname text;
del_job_tblname text := job_tblname || ‘_delete’;


BEGIN
  SELECT last_refreshed, refresh_interval, sql_delete, sql_insert 
  INTO last_r, r_interval, del_qry, ins_qry
  FROM d_log 
  WHERE table_name = tblname;

select last_sync 
into max_time
from sync_log where table_name = tblname

IF (last_r+r_interval) < max_time THEN

ins_qry := replace(ins_qry,‘fromdate’,quote_literal(last_r));
ins_qry := replace(ins_qry,‘todate’,quote_literal(last_r+r_interval));

del_qry := replace(del_qry,‘fromdate’,quote_literal(last_r));
del_qry := replace(del_qry,‘todate’,quote_literal(last_r+r_interval));

   del_stime := timeofday();
    execute del_qry;
    del_etime := timeofday();

    GET DIAGNOSTICS del_rows = ROW_COUNT;
    

    ins_stime := timeofday();
        BEGIN
	  execute ins_qry;
     	  EXCEPTION WHEN UNIQUE_VIOLATION THEN
	    execute d_refresh_delete(del_job_tblname);
	    execute ins_qry;
	END;
    ins_etime := timeofday();
 
    GET DIAGNOSTICS ins_rows = ROW_COUNT;

  UPDATE d_log 
  SET    SET last_refreshed = last_r + r_interval,
    record_update_date_time =  now(),
    delete_time = del_etime - del_stime,
    insert_time = ins_etime - ins_stime,
    rows_deleted = del_rows,
    rows_inserted = ins_rows
    WHERE job_table_name = job_tblname;
END IF;

RETURN;
END;
$BODY$
  LANGUAGE ‘plpgsql’ VOLATILE;

Function 2


CREATE OR REPLACE FUNCTION d_refresh_delete(tblname text)
  RETURNS void AS
$BODY$

DECLARE
last_r timestamp;
r_interval interval;
del_qry text;
ins_qry text;
del_stime timestamp;
del_etime timestamp;
ins_stime timestamp;
ins_etime timestamp;
max_time timestamp;
del_rows integer; 
ins_rows integer; 

tblname text;
base_tblname text := replace(job_tblname,‘_delete’,‘’);

BEGIN
   
   SELECT table_name, sql_delete, sql_insert 
  INTO tblname, del_qry, ins_qry
  FROM d_log 
  WHERE job_table_name = job_tblname;

  SELECT last_refreshed, refresh_interval
  INTO last_r, r_interval
  FROM d_log 
  WHERE job_table_name = base_tblname;


    ins_qry := replace(ins_qry,‘fromdate’,quote_literal(last_r));
    ins_qry := replace(ins_qry,‘todate’,quote_literal(last_r+r_interval));

    del_qry := replace(del_qry,‘fromdate’,quote_literal(last_r));
    del_qry := replace(del_qry,‘todate’,quote_literal(last_r+r_interval));

    del_stime := timeofday();
    execute del_qry;
    del_etime := timeofday();
    GET DIAGNOSTICS del_rows = ROW_COUNT;

    ins_stime := timeofday();
    execute ins_qry;
    ins_etime := timeofday();
    GET DIAGNOSTICS ins_rows = ROW_COUNT;

    UPDATE d_log 
    SET last_refreshed = last_r + r_interval,
    record_update_date_time =  now(),
    delete_time = del_etime - del_stime,
    insert_time = ins_etime - ins_stime,
    rows_deleted = del_rows,
    rows_inserted = ins_rows
    WHERE job_table_name = job_tblname;

RETURN;
END;
$BODY$
  LANGUAGE ‘plpgsql’ VOLATILE;

So.. when I call d_refresh(‘table_name’)
and a duplicate primary key condition occurs, it will call the d_refresh_delete(‘table_name_delete’) w/ the table name concatenated with a “_delete”.

In the d_refresh_delete function, it will first strip the “_delete” from the input tablename and use that to obtain the last_refresh and the refresh_interval from the base table so that it will use the same time/date/interval where the error occurs and then remove the duplicate into another duplicate table for further review.

Once finished, it will return the control back to the original originating function and continue again with the insert execution and the normal flow.

I’m pretty happy with this as it greatly reduces the amount of disk IO and server time. Don’t do useless stuffs right?

Plenty Cool.

SQL - My plpgSQL Function.. checking for maxtime

Sat, 20 Oct 2007 15:14:59 GMT

This is just an update on the function d_refresh(tblname text)

I wanted to ensure that the function will not run when the last sync time of the DB to the master DB is less than the last_refreshed time + refresh_interval time. So, I used an IF to wrap it up.

CREATE OR REPLACE FUNCTION d_refresh(tblname text)
  RETURNS void AS
$BODY$

DECLARE
last_r timestamp;
r_interval interval;
del_qry text;
ins_qry text;
del_stime timestamp;
del_etime timestamp;
ins_stime timestamp;
ins_etime timestamp;
max_time timestamp;

BEGIN
  SELECT last_refreshed, refresh_interval, sql_delete, sql_insert 
  INTO last_r, r_interval, del_qry, ins_qry
  FROM d_log 
  WHERE table_name = tblname;

select last_sync 
into max_time
from sync_log where table_name = tblname

IF (last_r+r_interval) < maxtime THEN

ins_qry := replace(ins_qry,‘fromdate’,quote_literal(last_r));
ins_qry := replace(ins_qry,‘todate’,quote_literal(last_r+r_interval));

del_qry := replace(del_qry,‘fromdate’,quote_literal(last_r));
del_qry := replace(del_qry,‘todate’,quote_literal(last_r+r_interval));

  del_stime := timeofday();
  execute del_qry;
  del_etime := timeofday();

  ins_stime := timeofday();
  execute ins_qry;
  ins_etime := timeofday();


  UPDATE d_log 
  SET last_refreshed = last_r + r_interval,
  record_update_date_time =  now(),
  delete_time = del_etime - del_stime,
  insert_time = ins_etime - ins_stime
  WHERE table_name = tblname;
END IF;

RETURN;
END;
$BODY$
  LANGUAGE ‘plpgsql’ VOLATILE;

Cool...

SQL - Optimising Delete

Sat, 20 Oct 2007 14:42:04 GMT

I admit, my sql foo still has to be honed.

My ~~previous~~ problem was that the previous entry’s profiling has found that I spent too much time on the delete portion of the query as opposed to the insert.

(what I was basically trying to do was just a simple DELETE then INSERT)

     table_name      |   delete_time   |   insert_time
---------------------+-----------------+-----------------
z                  | 00:01:14.424943 | 00:00:02.622862

the delete query was this..

select * from d_trz where exists(
select 1
from z
where z.record_update_date_time >= ‘2007-08-08 18:00:00’
and z.record_update_date_time   <  '2007-08-08 18:01:00'
and d_trz.id= z.id 
and d_trz.hid = z.hid 
and d_trz.start_date_time = z.start_date_time 
and d_trz.type = z.type 
and d_trz.phase_id = z.phase_id
)

the explain was giving me this:

Seq Scan on d_trz  (cost=0.00..414862.31 rows=21114 width=1611)"
  Filter: (subplan)"
  SubPlan"
    ->  Index Scan using idx_trz_uptime on z  (cost=0.00..9.71 rows=1 width=0)“
          Index Cond: ((record_update_date_time >= ‘2007-08-08 18:00:00’::timestamp without time zone) AND (record_update_date_time < '2007-08-08 18:01:00'::timestamp without time zone))"
          Filter: ((($0)::text = (id)::text) AND ($1 = hid) AND ($2 = start_date_time) AND ($3 = type) AND ($4 = phase_id))"

Performance really sucked when the number of rows in that table increases.

So, I went googling, but turned up nothing which will help me optimise the query. So.. Off I went to IRC and asked the question.
...
...
...

I got an answer (again) from akretschmerto try the query in an alternate way..


select * from d_trz where 
(id, hid, start_date_time, type, phase_id) in 
(select id, hid, start_date_time, type, phase_id from z
where z.record_update_date_time >= ‘2007-08-08 18:00:00’
and z.record_update_date_time   <  '2007-08-08 18:01:00') 

Nested Loop  (cost=9.71..18.05 rows=1 width=1611) (actual time=66.683..70.852 rows=82 loops=1)"
  ->  HashAggregate  (cost=9.71..9.72 rows=1 width=30) (actual time=66.634..67.047 rows=254 loops=1)”
        ->  Index Scan using idx_trz_uptime on z  (cost=0.00..9.70 rows=1 width=30) (actual time=0.107..9.729 rows=5170 loops=1)“
              Index Cond: ((record_update_date_time >= ‘2007-08-08 18:00:00’::timestamp without time zone) AND (record_update_date_time < '2007-08-08 18:01:00'::timestamp without time zone))"
  ->  Index Scan using d_trz_pkey on d_trz  (cost=0.00..8.30 rows=1 width=1611) (actual time=0.009..0.010 rows=0 loops=254)”
        Index Cond: (((d_trz.id)::text = (z.id)::text) AND (d_trz.hid = z.hid) AND (d_trz.start_date_time = z.start_date_time) AND (d_trz.type = z.type) AND (d_trz.phase_id = z.phase_id))“
Total runtime: 71.182 ms”

Many thanks again to akretschmer

SQL - Profiling a SQL function

Sat, 20 Oct 2007 09:25:55 GMT

It’s not really rocket science, but I wanted to know what was taking so long on my function. I’m not sure if it’s the time taken to delete (because the explain plan keeps doing a sequential scan as opposed to an index scan; then again the table is still very small and it’s now only 12K rows)

So.. I first proceeded to put in some “timers” in the function. However, (since I didn’t know better) I used current_timestamp and now().

The function looks something like this :-

CREATE OR REPLACE FUNCTION d_refresh(tblname text)
  RETURNS void AS
$BODY$

DECLARE
last_r timestamp;
r_interval interval;
del_qry text;
ins_qry text;
del_stime timestamp;
del_etime timestamp;
ins_stime timestamp;
ins_etime timestamp;

BEGIN
  SELECT last_refreshed, refresh_interval, sql_delete, sql_insert 
  INTO last_r, r_interval, del_qry, ins_qry
  FROM d_log 
  WHERE table_name = tblname;

ins_qry := replace(ins_qry,‘fromdate’,quote_literal(last_r));
ins_qry := replace(ins_qry,‘todate’,quote_literal(last_r+r_interval));

del_qry := replace(del_qry,‘fromdate’,quote_literal(last_r));
del_qry := replace(del_qry,‘todate’,quote_literal(last_r+r_interval));

  del_stime := now();
  execute del_qry;
  del_etime := now();

  ins_stime := now();
  execute ins_qry;
  ins_etime := now();


  UPDATE d_log 
  SET last_refreshed = last_r + r_interval,
  record_update_date_time =  now(),
  delete_time = del_etime - del_stime,
  insert_time = ins_etime - ins_stime
  WHERE table_name = tblname;

RETURN;
END;
$BODY$
  LANGUAGE ‘plpgsql’ VOLATILE;

But I then found out that the time returned was actually 0. Now, that most certainly can’t be right since the query (insert/delete) takes approx between 5 to 19 secs.

I pinged some guys on IRC #postgres and half an hour later, I found out my problem. Note that I used now() above. So, according to akretschmer

[akretschmer] now() and current_timestamp returns the start-timestamp of the current transaction
[akretschmer] timeofday() returns the exact time

so.. with that, the above becomes

CREATE OR REPLACE FUNCTION d_refresh(tblname text)
  RETURNS void AS
$BODY$

DECLARE
last_r timestamp;
r_interval interval;
del_qry text;
ins_qry text;
del_stime timestamp;
del_etime timestamp;
ins_stime timestamp;
ins_etime timestamp;

BEGIN
  SELECT last_refreshed, refresh_interval, sql_delete, sql_insert 
  INTO last_r, r_interval, del_qry, ins_qry
  FROM d_log 
  WHERE table_name = tblname;

ins_qry := replace(ins_qry,‘fromdate’,quote_literal(last_r));
ins_qry := replace(ins_qry,‘todate’,quote_literal(last_r+r_interval));

del_qry := replace(del_qry,‘fromdate’,quote_literal(last_r));
del_qry := replace(del_qry,‘todate’,quote_literal(last_r+r_interval));

  del_stime := timeofday();
  execute del_qry;
  del_etime := timeofday();

  ins_stime := timeofday();
  execute ins_qry;
  ins_etime := timeofday();


  UPDATE d_log 
  SET last_refreshed = last_r + r_interval,
  record_update_date_time =  now(),
  delete_time = del_etime - del_stime,
  insert_time = ins_etime - ins_stime
  WHERE table_name = tblname;

RETURN;
END;
$BODY$
  LANGUAGE ‘plpgsql’ VOLATILE;

and I get



MyDB=> select table_name, delete_time, insert_time from denorm_log where 
table_name = ‘z’;
     table_name      |   delete_time   |   insert_time
-----------------------+-----------------+-----------------
z                         | 00:00:02.105404 | 00:00:08.312243

string_to_array and getting a function to return table column_names

Sun, 07 Oct 2007 13:20:40 GMT

It’s true, I’m slow when it comes to PG in terms of writing plpgsql functions. I blame it on a lack of ample documentation. Seriously, when it comes to PG’s plpgsql documentation, it’s really sparse. I know there’s the official documentation and all, but to me, it’s not nearly as useful because it lacks examples and that sucks in a truly big way.

The only book I can find on PG is (what I have) a 1st edition PostgreSQL written by Korry Douglas and Susan Douglas. And that too has like perhaps a little over 30pages worth of stuffs on plpgsql docs.

In the Microsoft world, and the MySQL world and Oracle, there’s tons of books on their stored procedure. I myself have a few of those in my cupboard.

Anwyay, I digres. The point of this post is to let the reader know how to get all the column_names for a particular table returned as a string so that this can be used for perl-dbi functions to pull from mssql->pgsql automatically so that I don’t have to re-write the query to add additional columns when additional columns are added into the tables. This will make maintenance easier.

There are 2 ways of doing this. I went with the more complicated way 1st. (mainly because I didn’t know there was a simpler way) and I was banging my head against the wall for the better part of the day. (oh.. I did take an afternoon nap for 3 hours when my head hurt enough.)

Here’s the 1st one.



CREATE OR REPLACE FUNCTION select_columns(tablename text) RETURNS text as $$
DECLARE
   sql_str text;
   qry text;
BEGIN
	for sql_str in 
        select attname from pg_class 
	join pg_attribute 
	on pg_class.oid = pg_attribute.attrelid
	join pg_namespace
	on pg_namespace.oid = pg_class.relnamespace
	where relname = tablename
	and nspname = ‘myschema
	and attnum > 0
	LOOP
	qry := coalesce(qry || ’,‘,’‘) || sql_str;
	END LOOP;

   RETURN qry;
END;
$$
LANGUAGE plpgsql;

the


qry := coalesce(qry || ’,‘,’’) || sql_str;

is cool because it automatically strips out the comma from (before) the 1st column_name. (because in the original declaration, the qry is NULL)

This trick wouldn’t work if we had originally declared qry as


qry text:=‘’;

which makes qry not NULL

2nd method is definitely simpler and easier to be used and understood.


SELECT array_to_string(array(SELECT attname FROM pg_class
JOIN pg_attribute
ON pg_class.oid = pg_attribute.attrelid
JOIN pg_namespace
ON pg_namespace.oid = pg_class.relnamespace
WHERE relname = tablename
AND nspname = ‘myschema
AND attnum > 0), ’,‘);

I’ll be using the 2nd incarnation as my choice of query string.

Thanks to depesz in #Postgresql IRC for the pointers.

Extending LVM...

Sat, 06 Oct 2007 16:37:40 GMT

mrpotato ~ # pvcreate -t -v /dev/sdb1
Test mode: Metadata will NOT be updated.
Set up physical volume for “/dev/sdb1” with 35551398 available sectors
Zeroing start of device /dev/sdb1
Physical volume “/dev/sdb1” successfully created
Test mode: Wiping internal cache
Wiping internal VG cache

mrpotato ~ # pvscan
PV /dev/sda1 VG storage lvm2 [4.88 GB / 0 free]
PV /dev/sda4 VG storage lvm2 [4.26 GB / 0 free]
PV /dev/sdc1 VG storage lvm2 [16.95 GB / 32.00 MB free]
PV /dev/sdb1 lvm2 [16.95 GB]
Total: 4 [43.03 GB] / in use: 3 [26.08 GB] / in no VG: 1 [16.95 GB]

mrpotato ~ # pvdisplay
--- Physical volume ---
PV Name /dev/sda1
VG Name storage
PV Size 4.88 GB / not usable 0
Allocatable yes (but full)
PE Size (KByte) 4096
Total PE 1249
Free PE 0
Allocated PE 1249
PV UUID 8sI5z7-aaYp-mj1l-6q84-RJ04-6Xzq-1EydOK

--- Physical volume ---
PV Name /dev/sda4
VG Name storage
PV Size 4.26 GB / not usable 0
Allocatable yes (but full)
PE Size (KByte) 4096
Total PE 1090
Free PE 0
Allocated PE 1090
PV UUID m2pnY2-GQeg-Ajmo-NQ3W-6gpp-B41K-XPYIMi

--- Physical volume ---
PV Name /dev/sdc1
VG Name storage
PV Size 16.95 GB / not usable 0
Allocatable yes
PE Size (KByte) 4096
Total PE 4338
Free PE 8
Allocated PE 4330
PV UUID kuWduP-PURj-u7mZ-e2ws-Cq6F-S2tc-aEGk6e

--- NEW Physical volume ---
PV Name /dev/sdb1
VG Name
PV Size 16.95 GB
Allocatable NO
PE Size (KByte) 0
Total PE 0
Free PE 0
Allocated PE 0
PV UUID X96Ljl-J4Sv-GTSA-bznp-rQ8c-5Fj7-oUwu53

mrpotato ~ # vgdisplay
--- Volume group ---
VG Name storage
System ID
Format lvm2
Metadata Areas 3
Metadata Sequence No 14
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 3
Act PV 3
VG Size 26.08 GB
PE Size 4.00 MB
Total PE 6677
Alloc PE / Size 6669 / 26.05 GB
Free PE / Size 8 / 32.00 MB
VG UUID iMVDA3-T9m7-oiTy-6aR4-P82G-jGrd-rfJC5S

mrpotato ~ # vgextend -t -v storage /dev/sdb1
Test mode: Metadata will NOT be updated.
Checking for volume group “storage”
Test mode: Skipping archiving of volume group.
Adding physical volume ‘/dev/sdb1’ to volume group ‘storage’
Wiping cache of LVM-capable devices
Volume group “storage” will be extended by 1 new physical volumes
Test mode: Skipping volume group backup.
Volume group “storage” successfully extended
Test mode: Wiping internal cache
Wiping internal VG cache

mrpotato ~ # pvscan
PV /dev/sda1 VG storage lvm2 [4.88 GB / 0 free]
PV /dev/sda4 VG storage lvm2 [4.26 GB / 0 free]
PV /dev/sdc1 VG storage lvm2 [16.95 GB / 32.00 MB free]
PV /dev/sdb1 VG storage lvm2 [16.95 GB / 16.95 GB free]
Total: 4 [43.03 GB] / in use: 4 [43.03 GB] / in no VG: 0 [0 ]
mrpotato ~ # vgdisplay
--- Volume group ---
VG Name storage
System ID
Format lvm2
Metadata Areas 4
Metadata Sequence No 15
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 4
Act PV 4
VG Size 43.03 GB
PE Size 4.00 MB
Total PE 11016 <----
Alloc PE / Size 6669 / 26.05 GB
Free PE / Size 4347 / 16.98 GB
VG UUID iMVDA3-T9m7-oiTy-6aR4-P82G-jGrd-rfJC5S

mrpotato ~ # lvextend -l 11016 -t -v /dev/storage/storage
Test mode: Metadata will NOT be updated.
Finding volume group storage
Test mode: Skipping archiving of volume group.
Extending logical volume storage to 43.03 GB
Test mode: Skipping volume group backup.
Found volume group "storage"
Found volume group "storage"
Logical volume storage successfully resized
Test mode: Wiping internal cache
Wiping internal VG cache

rpotato ~ # e2fsck -f /dev/storage/storage
e2fsck 1.38 (30-Jun-2005)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/storage/storage: 1167/3384128 files (14.1% non-contiguous), 2674943/6829056 blocks
mrpotato ~ # resize2fs /dev/storage/storage
resize2fs 1.38 (30-Jun-2005)
Resizing the filesystem on /dev/storage/storage to 11280384 (4k) blocks.
The filesystem on /dev/storage/storage is now 11280384 blocks long.

http://kbase.redhat.com/faq/FAQ_85_4842.shtm
http://tldp.org/HOWTO/LVM-HOWTO/extendlv.htmlp

mrpotato ~ # emerge -av e2fsprogs

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild U ] sys-libs/com_err-1.40.2 [1.38] USE="nls" 3,873 kB
[ebuild U ] sys-libs/ss-1.40.2 [1.38] USE="nls" 0 kB
[ebuild U ] sys-fs/e2fsprogs-1.40.2 [1.38-r1] USE="nls -static" 0 kB

Total: 3 packages (3 upgrades), Size of downloads: 3,873 kB

Would you like to merge these packages? [Yes/No] n

With e2fsprogs >=1.39-1 new filesystems are created with directory indexing and
on-line resizing enabled by default (see /etc/mke2fs.conf).

mrpotato ~ # df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 6.9G 5.4G 1.5G 79% /
udev 505M 164K 505M 1% /dev
none 505M 0 505M 0% /dev/shm
/dev/mapper/storage-storage
43G 9.8G 33G 24% /storage

The Mantra of Successful BI

Sun, 23 Sep 2007 04:14:18 GMT

These days, my focus is geared towards learning and making SQL and in the past, I’ve tried to replicate from mssql to mysql and I wasn’t too successful in doing that due to various reasons.

However, now I’m learning and playing more and more with postgresql and I’m getting more and more impressed with it as a Database. While the setup differs between the company’s mssql server and mssql in terms of the number of columns (I’m using PG more like a datamart as opposed to a data warehouse), I’m also limited to lesser hardware w/ only a 2G celeron w/ 1G ram to be deployed on.

Anyway, that’s not the point of this post. The point of this post is to present to the larger community at hand on what makes a good BI Tool (Business Intelligence). BI is a hot topic these days what with everyone digging and mining through terabytes of raw or summarised data in search of the golden nugget. (I just read the papers today that the US Dept of Homeland Security is collecting and mining data on US residents to determine suspicious behaviour)

So, what makes a good BI tool? Point and Click? Drag and Drop? (yet another) New interface?

Let me tell you a story on what makes a good BI tool and something which (nearly) everyone can use w/o much re-learning. It’s a ubiquitus tool and though I’m largely a power user of it, I’m not recommending its use if you can utilise some other FOSS based versions. There’s nothing much to learn due to it’s ubiquitious nature and it’s pervasiveness.

You get your drag and drop and you get your point and click as well. It’s interface is user friendly and it provides a familiar surrounding to your users.

The tool, which I use and which I wrote macros for is called “EXCEL” (tm). Yep.. I (or rather we) use Microsoft(tm) Excel(tm) as a front end to the DB to obtain and to slice & dice the data and it’s working well.

Why Excel? I know there are lots of technical and non-technical reasons not to use Excel, but that’s besides the point. When you’re given lemon, you have to use it to make lemonade right? So, that was the situation provided to me and I grabbed it by it’s horns and make it work.

Adapt of die.

Reading Andy and his opinion on “The Price Of Failure” on the comment by Madan Sheina on the failure of BI projects.

I especially like and I quote Andy on point #3.

...

3. “Just one more new user interface” is not what the customer wants to hear. “Most are familiar with Excel and are not willing to change their business experience” was one quote from a customer in the article. Spot on! Why should a customer whose main job is, after all, not IT but something in the business, have to learn a different tool just to get access to data that he or she needs? Some tool vendors have done a good job of integrating with Excel, and yet are often in denial about this since they view their proprietary interface as a key competitive weapon against other vendors. Customers don’t care about this; they just want to get at the data they need to do their job on an easy and timely way. Hence a BI project should, if at all possible, look at ways of allowing users to getting data into their familiar Excel rather than foisting new interfaces on them. A few analyst types will be prepared to learn a new tool, but this is only a small subset of the audience for a BI project, likely 10% or less.

Like it or not, spreadsheets are here to stay.
Spreadsheets in any form, gnumeric, openoffice calc, excel, koffice anything which can connect to a DB and retrieve data is your greatest BI tool.

Rick Sherman, in a 2005 article wrote :

...
For many years BI vendors have been building front-end tools to try to replace spreadsheets for querying, reporting and analyzing data results. But despite the fact that tens of thousands of BI tool licenses have been sold, spreadsheets are still the most pervasive and dominant tool.
...

Forrester even went to publish a whitepaper entitled “Ouch! Get Ready - Spreadsheets are Here to Stay for Business Intelligence” which can be downloaded here for free

...
“Spreadsheets — the most widely used business intelligence (BI) tool — are a permanent fixture in enterprises because no other analytical application outperforms them in flexibility, ease of use, and ubiquity. Spreadsheets’ role in BI is no longer limited to simple import/export mechanisms; they now play an integral role in all layers of the BI stack. Yet the lack of controls and security and integrity issues create tremendous challenges. To minimize risks while gaining the inherent BI value of spreadsheets, information and knowledge management professionals must discriminate between the different ways spreadsheets are used. Then, they must help users apply advanced spreadsheet tools and techniques to their daily jobs, while also implementing a tightly controlled (or closely monitored) environment for critical production processes that rely on spreadsheet data. In turn, vendors should take advantage of this market opportunity by introducing tools that will bridge the gap between spreadsheet management and spreadsheet usage in the BI process.”
...

In the paper from Information Builders titled “Worst Practices in BI”, which can be downlaoded from here which has this bit of anecdote by Ralph Kimball

“The majority of the user base likely will access the data via pre -built parameter-driven analytic applications. Approximately 90 to 95 percent of the potential users will be served by these canned applications that are essentially finished templates that do not require users to construct relational queries directly.”

Bi tools needs to satisfy the needs of the main business users and it needs to provide these data in a timely fashion to gain the most from the “time to market” and to maintain competitiveness.

However, usage of spreadsheets is also a bane because of “operator error” as I will put it. One way to ensure that data is not calculated wrongly, is to ensure that users/operators need not apply formulas and other manual calculations themselves and just use the data as is. Excel or any other spreadsheet only provides the means (UI) for getting the data from the Data mart/warehouse for pivot tables.

One thing which I’ve yet to be able to do is to determine how to get what WebFOCUS does(you’ll have to refer to figure 2 & 3 of the report to understand) that is to also put the excel calculations into the spreadsheet when the data is exported.

That, is a cool feature.

So.. my friends, what BI tools are being used extensively in your environment? And Are your users using it properly? I know that I didn’t use the company’s new BI tool (actually, there was none since they pulled the plug on Business Objects) and asked users to render SQL themselves.