Josef “Jeff” Sipek

Jaunty Jackass

From IRC:

<jeffpc> Event: Happy Jaunty Release Day Everyone!
<jeffpc> Where: Corner Brewery
<jeffpc> Why: Because Ubuntu is awesome! Because we?re awesome! Right on.
* jeffpc groans
<jeffpc> obiwan: don't you ever take drugs, young man!
<jeffpc> obiwan: or you'll turn into one of those ubuntu lovers
<obiwan> hahahahahahahahhahahaahahahhaa
<obiwan> I was wondering why you cared about Jaunty Jackass or whatever the
         hell they call their releases
<jeffpc> hahaha
<jeffpc> that's awesome
<jeffpc> Jaunty Jackass
<obiwan> they just call it "Jaunty" like we're supposed to mentally fill in
         the other word
<obiwan> well there you go

SMTP

This is a (long overdue) reply to Ilya’s post: SMPT – Time to chuck it.

I’m going to quote it here, and reply to everything in it. Whenever I say "you," I mean Ilya. So, with that said, let’s get started.

E-mail, in particular SMTP (Simple Mail Transfer Protocol) has become an integral part of our lives, people routinely rely on it to send files, and messages. At the inception of SMTP the Internet was only accessible to a relatively small, close nit community; and as a result the architects of SMTP did not envision problems such as SPAM and sender-spoofing. Today, as the Internet has become more accessible, scrupulous people are making use of flaws in SMTP for their profit at the expense of the average Internet user.

Alright, this is pretty much the only thing I agree with.

There have been several attempts to bring this ancient protocol in-line with the current society but the problem of spam keeps creeping in. At first people had implemented simple filters to get rid of SPAM but as the sheer volume of SPAM increased mere filtering became impractical, and so we saw the advent of adaptive SPAM filters which automatically learned to identify and differentiate legitimate email from SPAM. Soon enough the spammers caught on and started embedding their ads into images where they could not be easily parsed by spam filters.

A history lesson…still fine.

AOL (America On Line) flirted with other ideas to control spam, imposing email tax on all email which would be delivered to its user. It seems like such a system might work but it stands in the way of the open principles which have been so important to the flourishing of the internet.

AOL (I believe Microsoft had a similar idea) really managed to think of something truly repulsive. The postal system in the USA didn’t always work the way it does today. A long time ago, the recipient paid for the delivery. AOL’s idea seems a lot like that.

There are two apparent problems at the root of the SMTP protocol which allow for easy manipulation: lack of authentication and sender validation, and lack of user interaction. It would not be difficult to design a more flexible protocol which would allow for us to enjoy the functionality that we are familiar with all the while address some, if not all of the problems within SMTP.

To allow for greater flexibility in the protocol, it would first be broken from a server-server model into a client-server model.

This is first point I 100% disagree with…

That is, traditionally when one would send mail, it would be sent to a local SMTP server which would then relay the message onto the next server until the email reached its destination. This approach allowed for email caching and delayed-send (when a (receiving) mail server was off-line for hours (or even days) on end, messages could still trickle through as the sending server would try to periodically resend the messages.) Todays mail servers have very high up times and many are redundant so caching email for delayed delivery is not very important.

"Delayed delivery is not very important"?! What? What happened to the whole "better late than never" idiom?

It is not just about uptime of the server. There are other variables one must consider when thinking about the whole system of delivering email. Here’s a short list; I’m sure I’m forgetting something:

  • server uptime
  • server reliability
  • network connection (all the routers between the server and the "source") uptime
  • network connection reliability

It does little to no good if the network connection is flakey. Ilya is arguing that that’s rarely the case, and while I must agree that it isn’t as bad as it used to be back in the 80’s, I also know from experience that networks are very fragile and it doesn’t take much to break them.

A couple of times over the past few years, I noticed that my ISP’s routing tables got screwed up. Within two hours of such a screwup, things returned to normal, but that’s 2 hours of "downtime."

Another instance of a network going haywire: one day, at Stony Brook University, the internet connection stopped working. Apparently, a compromised machine on the university campus caused a campus edge device to become overwhelmed. This eventually lead to a complete failure of the device. It took almost a day until the compromised machine got disconnected, the failed device reset, and the backlog of all the traffic on both sides of the router settled down.

Failures happen. Network failures happen frequently. More frequently that I would like them to, more frequently than the network admins would like them to. Failures happen near the user, far away from the user. One can hope that dynamic routing tables keep the internet as a whole functioning, but even those can fail. Want an example? Sure. Not that long ago, the well know video repository YouTube disappeared off the face of the Earth…well, to some degree. As this RIPE NCC RIS case study shows, on February 24, 2008, Pakistan Telecom decided to announce BGP routes for YouTube’s IP range. The result was, that if you tried to access any of YouTube’s servers on the 208.65.152.0/22 subnet, your packets were directed to Pakistan. For about an hour and twenty minutes that was the case. Then YouTube started announcing more granular subnets, diverting some of the traffic back to itself. Eleven minutes later, YouTube announced even more granular subnets, diverting large bulk of the traffic back to itself. Few dozen minutes later, PCCW Global (Pakistan Telecom’s provider responsible for forwarding the "offending" BGP announcements to the rest of the world) stopped forwarding the incorrect routing information.

So, networks are fragile, which is why having an email transfer protocol that allows for retransmission a good idea.

Instead, having direct communication between the sender-client and the receiver-server has many advantages: opens up the possibility for CAPTCHA systems, makes the send-portion of the protocol easier to upgrade, and allows for new functionality in the protocol.

Wow. So much to disagree with!

  • CAPTCHA doesn’t work
  • What about mailing lists? How does the mailing list server answer the CAPTCHAs?
  • How does eliminating server-to-server communication make the protocol easier to upgrade?
  • New functionality is a nice thing in theory, but what do you want from your mail transfer protocol? I, personally, want it to transfer my email between where I send it from and where it is supposed to be delivered to.
  • If anything eliminating the server-to-server communication would cause the MUAs to be "in charge" of the protocols. This means that at first there would be many competing protocols, until one takes over - not necessarily the better one (Betamax vs. VHS comes to mind).
  • What happens in the case of overzealous firewall admins? What if I really want to send email to bob@example.com, but the firewall (for whatever reason) is blocking all traffic to example.com?

Spam is driven by profit, the spammers make use of the fact that it is cheap to send email. Even the smallest returns on spam amount to good money. By making it more expensive to send spam, it would be phased out as the returns become negative. Charging money like AOL tried, would work; but it is not a good approach, not only does it not allow for senders anonymity but also it rewards mail-administrators for doing a bad job (the more spam we deliver the more money we make).

Yes, it is unfortunately true, money complicates things. Money tends to be the reason why superior design fails to take hold, and instead something inferior wins - think Betamax vs. VHS. This is why I think something similar would happen with competing mail transfer protocols - the one with most corporate backing would win, not the one that’s best for people.

Another approach is to make the sender interact with the recipient mail server by some kind of challenge authentication which is hard to compute for a machine but easy for a human, a Turing test. For example the recipient can ask the senders client to verify what is written on an obfuscated image (CAPTCHA) or what is being said on a audio clip, or both so as to minimize the effect on people with handicaps.

Nice thought about the handicapped, but you are forgetting that only 800-900 million people speak English (see Wikipedia article: Wikipedia). That is something on the order of 13-15 percent. Sorry, but "listening comprehension" tests are simply not viable.

Obfuscated image Wikipedia article: CAPTCHAs are "less" of a problem, but then again, one should consider the blind. I am not blind, and as a matter of fact my vision is still rather good (even after years of staring at computer screens), but at times I’m not sure what those "distorted text" CAPTCHAs are even displaying. I can’t even begin to imagine what it must be like for anyone with poor vision.

You seem to be making the assumption that most if not all legitimate email comes from humans. While that may be true for your average home user, let’s not forget that email is used by more technical people as well. These people will, and do, use email in creative ways. For example, take me…I receive lots of emails that are generated by all sorts of scripts that I wrote over time. These emails give me status of a number of systems I care about, and reminders about upcoming events. All in all, you could say that I live inside email. You can’t do a CAPTCHA for the process sending the automated email (there’s no human sending it), and if you do the CAPTCHA for the receiving, you’re just adding a "click here to display the message" wart to the mail client software user interface.

Just keep in mind that all those automated emails you get from "root" or even yourself were sent without a human answering a CAPTCHA.

It would be essential to also white list senders so that they do not have to preform a user-interactive challenge to send the email, such that mail from legitimate automated mass senders would get through (and for that current implementation of sieve scripts could be used). In this system, if users were to make wide use of filters, we would soon see a problem. If nearly everyone has a white list entry for Bank Of America what is to prevent a spammer to try to impersonate that bank?

White listing is really annoying, and as you point out, it doesn’t work.

And so this brings us to the next point, authentication, how do you know that the email actually did, originate from the sender. This is one of the largest problems with SMTP as it is so easy to fake ones outgoing email address. The white list has to rely on a verifiable and consistent flag in the email. A sample implementation of such a control could work similar to the current hack to the email system, SPF, in which a special entry is made in the DNS entry which says where the mail can originate from. While this approach is quite effective in a sever-server architecture it would not work in a client-server architecture. Part of the protocol could require the sending client to send a cryptographic-hash of the email to his own receiving mail server, so that the receiving party’s mail server could verify the authenticity of the source of the email. In essence this creates a 3 way handshake between the senders client, the senders (receiving) mail server and the receiver’s mail server.

I tend to stay away from making custom authentication protocols.

In this scheme, what guarantees you that the client and his "home server" aren’t both trying to convince the receiving server that the email is really from whom they say it is? In kerberos, you have a key for each system, and a password for each user. The kerberos server knows it all, and this central authority is why things work. With SSL certificates, you rely on the strength of the crypto used, as well as blind faith in the certificate authority.

At first it might seem that this process uses up more bandwidth and increases the delay of sending mail but one has to remember that in usual configuration of sending email using IMAP or POP for mail storage one undergoes a similar process,

Umm…while possible, I believe that very very large majority of email is sent via SMTP (and I’m not even counting all the spam).

first email is sent for storage (over IMAP or POP) to the senders mail server and then it is sent over SMTP to the senders email for redirection to the receivers mail server. It is even feasible to implement hooks in the IMAP and POP stacks to talk to the mail sending daemon directly eliminating an additional socket connection by the client.

Why would you want to stick with IMAP and POP? They do share certain ideas with SMTP.

For legitimate mass mail this process would not encumber the sending procedure as for this case the sending server would be located on the same machine as the senders receiving mail server (which would store the hash for authentication), and they could even be streamlined into one monolithic process.

Not necessarily. There are entire businesses that specialize in mailing list maintenance. You pay them, and they give you an account with software that maintains your mailing list. Actually, it’s amusing how similar it is to what spammers do. The major difference is that in the legitimate case, the customer supplies their own list of email address to mail. Anyway, my point is, in these cases (and they are more common than you think) the mailing sender is on a different computer than the "from" domain’s MX record.

Some might argue that phasing out SMTP is a extremely radical idea, it has been an essential part of the internet for 25 years.

Radical? Sure. But my problem is that there is no replacement. All the ideas you have listed have multiple problems - all of which have been identified by others. And so here we are, no closer to the solution.

But then, when is the right time to phase out this archaic and obsolete protocol, or do we commit to use it for the foreseeable future. Then longer we wait to try to phase it out the longer it will take to adopt something new. This protocol should be designed with a way to coexist with SMTP to get over the adoption curve, id est, make it possible for client to check for recipients functionality, if it can accept email by this new protocol then send it by it rather than SMTP.

Sounds great! What is this protocol that would replace SMTP? Oh, right there isn’t one.

The implementation of such a protocol would take very little time, the biggest problem would be with adoption.

Sad, but true.

The best approach for this problem is to entice several large mail providers (such as Gmail or Yahoo) to switch over. Since these providers handle a large fraction of all mail the smaller guys (like myself) would have to follow suit.

You mentioned Gmail…well, last I heard, Gmail’s servers were acting as open proxies. Congratulations! One of your example "if they switch things will be better" email providers is allowing the current spam problem to go on. I guess that makes you right. If Gmail were to use a protocol that didn’t allow for spam to exist, then things would be better.

There is even an incentive for mail providers to re-implement mail protocol, it would save them many CPU-cycles since Bayesian-spam-filters would no longer be that important.

What about generating all those CAPTCHAs you suggested? What about hashing all those emails? Neither activity is free.

By creating this new protocol we would dramatically improve an end users experience online, as there would be fewer annoyances to deal with. Hopefully alleviation of these annoyances will bring faster adoption of the protocol.

I really can’t help but read that as "If we use this magical protocol that will make things better, things will get better!" Sorry, but unless I see some protocol which would be a good candidate, I will remain sceptical.

As a side note, over the past  90 days, I received about 164MB of spam that SpamAssassin caught and procmail promptly shoved into the spam mail box. Do I care? Not enough to jump on the "let’s reinvent the email system" bandwagon. Sure, it eats up some of my servers clock cycles, and some bandwidth, the spam that gets to me is the few pieces that manage to get through, and show up in my inbox. Would I be happy if I didn’t have to use a spam filter, and not have to delete the few random spams by hand? Sure, but at least for the moment, I don’t see a viable alternative.

O_PONIES & Other Assorted Wishes

You might have already heard about ext4 “eating” people’s data. That’s simply not true.

While I am far from being a fan of ext4, I feel an obligation to set the record straight. But first, let me give you some references with an approximate timeline. I’m sure I managed to leave out a ton of details.

In mid-January, a bug titled Ext4 data loss showed up in the Ubuntu bug tracker. The complaining users apparently were using data on system crashes when using ext4. (The fact that Ubuntu likes to include every unstable & crappy driver into their kernels doesn’t help at all.) As part of the discussion, Ted Ts’o explained that the problem wasn’t with ext4 but with applications that did not ensure that the data they wrote was actually safe. The people did not like hearing that.

Things went pretty quiet until mid-March. That’s when a slashdot article made it painfully obvious that many of today’s apps are buggy. Some applications (KDE being a whole suite of applications) gotten used to the fact that ext3 was a very common filesystem used by Linux installations. More specifically, they got used to the behavior that ext3’s default mount option (data=ordered) provided. This is really the issue. The application developers assumed that the POSIX interface gave them more guarantees that it did! To make matters worse, the one way to ensure that the contents of a file get to the disk (the fsync system call) is very expensive on ext3. So over the past (almost) decade that ext3 has been around, application developers have been “trained” (think Wikipedia article: Pavlov reflexes) to not use fsync — on ext3, it’s expensive and the likelyhood of you losing data is much lower due to the default mount options. ext4’s fsync implementation, much like other filesystems’ implementations (e.g., XFS) does not suffer from this. (You may have heard about fsync on ext3 being expensive almost a year ago when Firefox was hit by this: Fsyncers and curveballs (the Firefox 3 fsync() problem). Note that in this case, as Ted Ts’o points out, the problem is that Firefox uses the same thread to draw the UI and do IO. That’s plain stupid.)

Over the next few days, Ted Ts’o posted two blog entries about delayed allocation (people seem to like to blame it for dataloss): Delayed allocation and the zero-length file problem, Don’t fear the fsync!.

About the same time, Eric Sandeen wrote a blurb about the state of affairs: fsync, sigh. He points out that XFS has faced the same issue years ago. When the application developers were confronted about their application being broken, they just put fingers in their ears, hummed loudly, yelled “I can’t hear you!” There is a word for that, and here’s the OED definition for it:

denial,

The asserting (of anything) to be untrue or untenable; contradiction of a statement or allegation as untrue or invalid; also, the denying of the existence or reality of a thing.

The problem is application developers not wanting to believe that it’s an application problem. Well, it really is! Not only are those apps broken, but they are not portable. AIX, IRIX, or Solaris will not give you the same guarantees as ext3!

(Eric is also trying to fight the common misconception that XFS nulls files: XFS does not null files, and requires no flux, which I assure you is not the case.)

About a week later, on an episode of Free Software Round Table, the problem was discussed a bit. They got most of it right :) (Here’s a 55MB mp3 of the show: 2009-03-21.)

When April 1st came about, the linux-fsdevel mailing list got a patch from yours truly: [PATCH] fs: point out any processes using O_PONIES. (The pony thing…it’s a bit of an inside joke among the Linux filesystem developers.) The idea of having O_PONIES first came up in #linuxfs on OFTC. While I don’t remember who first thought of it (my guess would be Eric), I know for sure that it wasn’t me. At the same time, I couldn’t help it, and considering that the patch took only a minute to make (and compile test), it was well worth it.

Few days later, during the Linux Storage and Filesystem workshop, the whole fsync issue got some discussion time. (See “Rename, fsync, and ponies” at Linux Storage and Filesystem workshop, day 1.) The part that really amused me:

Prior to Ted Ts’o’s session on fsync() and rename(), some joker filled the room with coloring-book pages depicting ponies. These pages reflected the sentiment that Ted has often expressed: application developers are asking too much of the filesystem, so they might as well request a pony while they’re at it.

In the comments for that article you can find Ted Ts’o saying:

Actually, it was Josef ’Jeff’ Sipek who deserves the first mention of application programmers asking for pones, when he posted an April Fools patch submission for the new open flag, O_PONIES — unreasonable file system assumptions desired.

Another file system developer who had worked on two major filesystems (ext4 and XFS) had a t-shirt on that had O_PONIES written on the front. And the joker who distributed the colouring book pages with pictures of ponies was another file system developer working yet another next generation file system.

Application programmers, while they were questioning my competence, judgement, and even my paternity, didn’t quite believe me when I told them that I was the moderate on these issues, but it’s safe to say that most of the file system developers in the room were utterly unsympathetic to the idea that it was a good idea to encourage application programmers to avoid the use of fsync(). About the only one who was also a moderate in the room was Val Aurora (formerly Henson). Both of us recognize that ext3’s data=ordered mode was responsible for people deciding that fsync() was harmful, and I’ve said already that if we had known how badly it would encourage application writers to Do The Wrong Thing, I would have pushed hard not to make data=ordered the default. Unfortunately, memory wasn’t as plentiful in those days, and so the associated page writeback latencies wasn’t nearly as bad ten years ago.

Hrm, I’m not sure how to take it…he makes it sound like I’m an extremist. Jeff — a freedom fighter for sanity of filesystem interfaces! :) As I said, I can’t take credit for the idea of O_PONIES. As I was writing this entry, I mentioned it to Eric and he promptly wrote an entry of his own: Coming clean on O_PONIES. It looks like he isn’t sure that he was the one to invent it! I’ll give him credit for it anyway.

The next day, a group photo of the attendees was taken… You can clearly see Val Aurora wearing an O_PONIES shirt. The idea was Eric’s, and as far as I know, he had his shirt the first day.

Fedora 11 is supposedly going to use ext4 as the default filesystem. When Ars Technica published an article about it (First look: Fedora 11 beta shows promise), some misguided people thinking that that ext4 eats your data left a bunch of comments….*sigh*

Well, there you have it. That’s the summary of events with some of my thoughts interleaved. If you are writing a userspace application that does file IO, do the right thing, fsync the data you care about (or at least fdatasync).

Blowfish on 24

I can’t belive this…These TV show writers really have to be trying to get things wrong.

Designing an Authentication System: a Dialogue in Four Scenes

I just learned about a rather non-technical description of the Kerberos authentication service: Designing an Authentication System: a Dialogue in Four Scenes. It makes me feel like I should set up Kerberos on my boxes…not sure if it’s even worth it, considering that I deal with no more than 3 computers (laptop, desktop, server).

Compare and Swap Manifesto

The fall semester, I was taking a computer architecture course, and during it, I came to the conclusion that compare-and-swap is better than the load-locked/store-conditional combination some architectures offer instead. As the class was very Alpha ISA-centric, the below text reflects that.

As a result I wrote this manifesto. I see no reason to keep it to myself! So, here it is. (Note: I haven’t really read though it in a couple of months, and overall, I didn’t take too long to write it, so there might be some nasty corner cases.)

Definition of Compare and Swap

Let us assume for the moment, that the compare and swap instruction is:

        CS      R1,R2,D3(R3)

Where, Rn specifies a register used as the n-th operand, and Dn specifies an immediate used as a displacement in the n-th operand.

The first and third operands are compared. If they are equal, the second operand is stored at the third-operand location and the first operand is set to the value zero. If they are unequal, the first operand is set to the value one.

When an equal comparison occurs, the second operand is stored at the third-operand location and the first operand is set to the value zero. The fetch of the third operand for purposes of comparison and the store into the third-operand location appear to be a block-concurrent interlocked-update reference as observed by other CPUs.

When the result of the comparison is unequal, the first operand is set to the value one, and the second-operand location remains unchanged.

Sample Code

This section illustrates how certain atomic primitives can be implemented with compare and swap. They assume Alpha-style instruction set and ignore Alpha’s weak memory ordering.

spinlock_lock:
        lda     r4,0x0
        lda     r5,0x1
        cs      r4,r5,lock_var
        bne     r4,spinlock_lock

spinlock_unlock:
        stq     r31,lock_var

atomic_<op>:
        ldq     r4,atomic_var
        <some operations that yield r5, and do not modify r4>
        cs      r4,r5,atomic_var
        bne     r4,atomic_<op>

Instruction Encoding

So far, we have ignored all aspects of how a compare and swap instruction may be encoded.

Before looking at the 4 instruction formats present in Alpha, let us consider the requirements of compare and swap.

Compare and swap has three operands:

  • the expected ("old") value
  • the new value
  • memory address

Additionally, it also returns a zero or one value in the first operand. Obviously, the instruction requires a minimum of three registers. While memory references are traditionally represented as <base-register, displacement> pairs (or as <base-reg, index-reg, displacement>, or even <base-reg, index-reg, scale-factor, displacement>) only a single register is necessary to specify an arbitrary memory location. Of the 4 instruction formats present on Alpha, only the Operate Instruction Format can encode 3 registers.

        +--------+----+----+---------+----+
        | Opcode | Ra | Rb | /////// | Rc |
        +--------+----+----+---------+----+
         31     26   21   16         5    0

Not having an immediate value to use as a displacement is unfortunate, however it does not make it impossible to implement compare and swap.

The actual mapping of the 3 registers to the Ra, Rb, and Rc instruction fields trivial, and can be essentially arbitrary. It is worth noting that the mapping should try to stay consistent with the other Operate format instructions. Namely, Rc should encode the destination register. For compare and swap, the only register that gets written to is the first operand (R1 in the example from the definition section). The order of the remaining two register mapping can be arbitrary with little to no effect on decoder complexity.

There are many unused opcodes that could be used for a compare and swap without adding any significant complexity to the decoder logic.

It should be noted that the bits 15:6 can be used as a displacement. The Op instruction format specifies that bit 12, must be zero if the instruction uses three register, or one if it uses two registers and an 8-bit immediate.

Potential Microarchitectural Issues

Even though only three registers are specified, compare and swap essentially uses 4 operands. The "additional" operand is the result of the fact that the first operand is first read, and then at a later time, written to.

This dictates that the Register Address Table (or some other register renaming logic) must handle translating three architected register numbers to physical register numbers, as well as allocating a new physical register number for the first operand. Aside from the RAT, other structures may need additional read ports (e.g., the physical register file must read 3 values concurrently).

Since compare and swap is a memory operation, special care must be taken to handle any interactions with a load/store queue. Given the fact that compare and swap potentially stores to memory, a mechanism must be provided to either not speculate (e.g., not begin execution of the compare and swap instruction until there are no proceeding instructions in the reorder buffer), or to speculatively launch the instruction and have an undo mechanism in case a conflicting store (from within the same core, or from another core) occurs. Obviously, it is possible to allow only a certain degree of speculation.

Another interesting question is, what component does the actual "compare and swap". The comparison could be located within the core, within the L1, or potentially somewhere else. Each has its own set of benefits and drawbacks.

in-core

This allows the caches to remain (relatively) simple. The cache needs to support a regular load, a regular store, and a "conditional" store.

The conditional store functions as follows:

  1. core issues a load
  2. core compares the loaded value with expected value
  3. if mis-match, destination register is set to 1
  4. if match, tell the cache to do a conditional store
  5. the cache takes the conditional store, and if the cache line was not invalidated since the load, store the new value; at the same time, signal the core informing it whether or not the line was invalidated. This bit is then used to determine the value stored in the destination register.

in-cache

This design makes the cache do the comparison. The advantage is that the cache does not need to send the data to the core, wait for the store/don’t-store signal, and then tell the core whether or not the store succeeded. Instead, it gets the expected value, does the comparison itself, and if the value matched, it proceeds with the store. Then, it signals the core, specifying whether or not the store occurred.

The cache can obtain the line in the exclusive state to avoid having to go to the bus for the store portion of the operation.

Compare and Swap vs. Load-locked/Store-conditional

Compare and swap has the interesting property of not caring whether or not the memory got changed. It simply does a comparison to determine if the in-memory value is what it expects.

Load-locked/Store-conditional lacks this. On the other hand, Store-conditional does not have to even attempt to access memory if the in-core lock register does not match the address.

References

[1] Alpha Architecture Handbook, version 4. EC–QD2KC–TE. Compaq
[2] z/Architecture Principles of Operation. page 7-66. SA22-7832-06. IBM

HVF: Sample Session

Looking at some of my older posts about z/Architecture, I decided to post a sample console session (including some annotations) with the latest version of the code with some work-in-progress patches that I haven’t touched in a while.

Every OS needs a nice banner, right?

                    HH        HH  VV        VV  FFFFFFFFFFFF
                    HH        HH  VV        VV  FFFFFFFFFFFF
                    HH        HH  VV        VV  FF
                    HH        HH  VV        VV  FF
                    HH        HH  VV        VV  FF
                    HHHHHHHHHHHH  VV        VV  FFFFFFF
                    HHHHHHHHHHHH  VV        VV  FFFFFFF
                    HH        HH   VV      VV   FF
                    HH        HH    VV    VV    FF
                    HH        HH     VV  VV     FF
                    HH        HH      VVVV      FF
                    HH        HH       VV       FF

HVF VERSION v0.15-rc4-7-g62eac50

NOW 06:38:44 UTC 2009-04-15

LOGON AT 06:38:45 UTC 2009-04-15

IPL command isn’t completely done, so for the time being, It has the device number hardcoded in.

ipl
WARNING: IPL command is work-in-progress
GUEST IPL HELPER LOADED; ENTERED STOPPED STATE

You can see the device number in R2, the SSCH in R1, and the base address in R12.

d g
GR  0 = 0000000000000000 0000000000010005
GR  2 = 0000000000000a00 0000000000000000
GR  4 = 0000000000000000 0000000000000000
GR  6 = 0000000000000000 0000000000000000
GR  8 = 0000000000000000 0000000000000000
GR 10 = 0000000000000000 0000000000000000
GR 12 = 0000000001000000 0000000000000000
GR 14 = 0000000000000000 0000000000000000

Execution will begin at 16MB, that’s where the loader gets copied.

d psw
PSW = 00080000 81000000 00000000 00000000

The first few instruction of the loader…as disassembled by the built in disassembler.

d s i1000000.20
R0000000001000000  B234C090      STSCH  144(R12)
R0000000001000004  4770C040      BC     7,64(R0,R12)
R0000000001000008  9680C095      OI     149(R12),128
R000000000100000C  B232C090      MSCH   144(R12)
R0000000001000010  4770C040      BC     7,64(R0,R12)
R0000000001000014  D2070078C060  MVC    120(8,R0),96(R12)
R000000000100001A  5830007C      L      R3,124(R0,R0)
R000000000100001E  4133C03C      LA     R3,60(R3,R12)

There are real devices. Since this run was under Hercules, these were all defined in the hvf.cnf.

q real
CPU RUNNING
STORAGE = 128M
CONS 0009 3215 SCH = 10000
RDR  000C 3505 SCH = 10001
PUN  000D 3525 SCH = 10002
PRT  000E 1403 SCH = 10003
GRAF 0040 3278 SCH = 10004
GRAF 0041 3278 SCH = 10005
TAPE 0580 3590 SCH = 10006

And there are virtual devices (including their subchannel information blocks).

q virtual
CPU STOPPED
STORAGE = 17M
CONS 0009 3215 ON CONS 0009 SCH = 10000
RDR  000C 3505 SCH = 10001
PUN  000D 3525 SCH = 10002
PRT  000E 1403 SCH = 10003
DASD 0191 3390      0 CYL ON DASD 0000 SCH = 10004
d schib all
SCHIB DEV  INT-PARM ISC FLG LP PNO LPU PI MBI  PO PA CHPID0-3 CHPID4-7
10000 0009 00000000   0  01 80  00  00 80 —- FF 80 00000000 00000000
10001 000C 00000000   0  01 80  00  00 80 —- FF 80 00000000 00000000
10002 000D 00000000   0  01 80  00  00 80 —- FF 80 00000000 00000000
10003 000E 00000000   0  01 80  00  00 80 —- FF 80 00000000 00000000
10004 0191 00000000   0  01 80  00  00 80 —- FF 80 00000000 00000000

Let ’er rip! Well, it gets past SSCH (well, kind of) and then it stopped when it didn’t know what to do with a DIAG.

be
INTRCPT: INST (b234 c0900000)
STSCH handler (raw:0000b234c0900000 addr:0000000001000090 sch:10005)
INTRCPT: INST (8300 00010000)
Unknown/mis-handled intercept code 04, err = -4

Ah, condition code 3, that’s why the loader gave up with DIAG, instead of attempting MSCH.

d psw
PSW = 00083000 81000048 00000000 00000000
d s i1000040.10
R0000000001000040  980FC0C4      LM     R0,R15,196(R12)
R0000000001000044  83000001      DIAG   X’000001’
R0000000001000048  980FC0C4      LM     R0,R15,196(R12)
R000000000100004C  83000000      DIAG   X’000000’

What version is this anyway? Is it 6:45 already?!

q cplevel
HVF version v0.15-rc4-7-g62eac50
IPL at 06:38:44 UTC 2009-04-15
q time
TIME IS 06:45:26 UTC 2009-04-15

P.S. I just realized that the post id for this post is 360. How apt! :)

HOWTO: Installing CentOS 4.x under z/VM

I guess I should mention it here…

Almost 6 months ago, I wrote up another howto: Installing CentOS 4.x under z/VM (the first one being Installing Debian under Hercules).

Goldman Sachs

I just came across an article that points out how Goldman Sachs managed to show a fair amount of profit for the first quarter:

Goldman Sachs reported a profit of $1.8 billion in the first quarter, and plans to sell $5 billion in stock and get out of the government’s clutches, if it can.

How did it do that? One way was to hide a lot of losses in not-so-plain sight.

Goldman’s 2008 fiscal year ended Nov. 30. This year the company is switching to a calendar year. The leaves December as an orphan month, one that will be largely ignored. In Goldman’s earnings statement, and in most of the news reports, the quarter ended March 31 is compared to the quarter last year that ended in February.

The orphan month featured “surprise” lots of write-offs. The pretax loss was $1.3 billion, and the after-tax loss was $780 million.

Would the firm have had a profit if it had stuck to its old calendar, and had to include December and exclude March?

Clever, eh?

Leviathan

Recently, I decided to watch a movie of some sort. Well, I ended up seeing Leviathan from 1989. It’s truly a bad movie.

Spoiler follows!

Leviathan

So, it’s essentially just like the movie Alien from 1979, except it takes place on the bottom of the ocean. You get the same kind of "monster" creature that’s impossible to kill, and it wants to kill everyone.

It starts off when the deep-ocean mining company workers find a ship-wreck of a ship that’s hasn’t (officially) sunk. Since the ship was Russian, they discover some vodka in the captains safe. Some of the crew consumes it. Little did they know that the vodka contains some virus engineered by the Russian military! (Everyone know that the Russian military experimented on their own men, right?) Anyway, these charismatic non-Russians’ DNA is now being rewritten by this virus. They are turning into…something!

After some battling (I’m not quite sure why they had a flame thrower on their ocean-floor base) of this "creature," they realize that they are on their own. The corporate headquarters don’t want to scare investors away with this "virus that rewrites your DNA" news, so they essentially wanted to let them die on the ocean floor. Little did the corporate heads know that our heroes are not only in deep s*..err..water, but also very creative and so they find a way to destroy the mutant, and at the same time save their rears.

So out of the 5 or 6 characters that we start with, all but two die. Oh, actually, one of the characters dies because a shark gets him! Doesn’t that bite (hahaha)? You are trying to escape a weird mutant creature that is a mix of your former buddies, and a shark gets you in the last 2 minutes of the movie!

If you have seen Alien, you’ve seen Leviathan. It really is that simple.

Powered by blahgd