Eric Pierce, University of South Florida High Availability Basics Before Clustering CAS Failover with Heartbeat Ticket Registry Load Balancing CAS at USF Make a list of possible Single-Points-of-Failure Single connections to ANYTHING (Power, Network, etc) Not just your servers – think about the datacenter Try to quantify for management How likely is this failure? If it happens, how long will it take to fix? How much will we lose while it is down? Don’t forget the human element! Make a list of possible solutions There are multiple ways to combat most SPoFs Assign a relative cost score to each The scoring system depends on your resources Some things are easy to implement, but expensive Cheaper solutions are (usually) more time-consuming Work with management What risks are they willing to accept? CAS is the central hub to all your web applications Without CAS, no one can use any applications A single machine is not enough CAS Architecture Authentication Service Management and Auditing Authentication Source Active Directory Multiple Domain Controllers LDAP replication Multi-Master replication Kerberos JAAS can query multiple KDCs Database Replication abilities product-specific Service Management Storage Options Database LDAP Service Registry is reloaded on all cluster nodes on a regular basis (since 3.3.4) Auditing & Statistics Storage Options Database Local File Both are optional, but recommended for production Heartbeat Failover versus Load Balancing Part of the Linux-HA Project Runs on most Unix-based Operating Systems Provides communication layer between cluster nodes Sends regular ‘heartbeat’ between nodes to test health Cluster Resource Manager handles starting/stopping resources CRM from Heartbeat has spun-off to a separate project: Pacemaker - http://clusterlabs.org Very easy to configure Linux distros include all you need GUI and CLI clients for setup & management No changes to CAS configuration required User Experience All TGTs & STs are lost on failover Users must re-authenticate after failover Wasted Resources If both servers are up, one is totally idle Resource Usage improves Both servers are now utilized 100% of the time Hardware SSL on the LB might improve performance User Experience is worse Half (on average) of all ticket verifications fail The TicketRegistry is not shared between servers Resource Usage improves Both servers are now utilized 100% of the time Hardware SSL on the LB might improve performance User Experience is worse Half (on average) of all ticket verifications fail The TicketRegistry is not shared between servers JBOSS Cache Memcached Java Persistence API Clustered cache service Distributes cache changes using JGroups Cache storage is not persistent in default config JDBC and flat-file storage available for persistence Details on setting up JBossCacheTicketRegistry are available at the Jasig Wiki: http://www.ja-sig.org/wiki/display/CASUM/Clustering+CAS JBOSS Cache JBOSS Cache JBOSS TicketRegistry Create Ticket Create Ticket Read Ticket Ticket Ticket Read Ticket JBOSS Cache Ticket Create Ticket Read Ticket Distributed caching system Hashing algorithm selects which node to store data on Cache is stored in memory Cache storage is not persistent Oldest objects are removed when cache is filled Simple, lightweight and fast Repcached patch adds 2-server data replication http://repcached.lab.klab.org/ Project stagnate? Memcached TicketRegistry Memcached Ticket Foo Memcached Ticket Bar Ticket Baz Hash Results Server 2 Server 1 Baz = Server 2 Bar = Server 2 Foo = Server 1 Set Set Memcached TicketRegistry Memcached Ticket Foo Memcached Ticket Bar Ticket Baz Hash Results Server 2 Server 1 Baz = Server 2 Bar = Server 2 Foo = Server 1 Set Set Repcache Ticket Foo Ticket Bar Ticket Baz Foo = Server 1 Bar = Server 1 Baz = Server 1 Tickets are stored in a database Storage is persistent Database HA is a necessity! Performance is can be very good Dependant on the speed of the db configuration Registry Cleaning Deadlocks have been an issue with the default cleaner CAS 3.4 introduces LockingStrategy Memcached TicketRegistry Memcached Ticket Foo Memcached Ticket Bar Ticket Baz Hash Results Server 2 Server 1 Baz = Server 2 Bar = Server 2 Foo = Server 1 Set Set Repcache Ticket Foo Ticket Bar Ticket Baz Foo = Server 1 Bar = Server 1 Baz = Server 1 Tickets are stored in a database Storage is persistent Database HA is a necessity! Performance is can be very good Dependant on the speed of the db configuration Registry Cleaning Deadlocks have been an issue with the default cleaner CAS 3.4 introduces LockingStrategy Cleaner attempts to ensure exclusive access to the DB before removing any expired tickets Uses a database table to hold lock state Only one node can clean the registry at a time Lock can be set by any node after expiration time JBoss Cache Very flexible but complicated Good option for clusters >2 nodes Memcached Easiest option for a 2-node cluster Status of repcache project is a concern JPA Best data integrity/reliability Obvious choice if you already have an HA database Best choice for very long ticket lifetimes (Remember me) Needs CAS 3.3.4 or newer (3.4 would be best) Load Balancing with Free software Hardware vs. Software Load Balancing N-to-N Cluster Combination of Apache modules mod_proxy_ajp mod_proxy_balancer Simple to configure: ProxyPass /cas balancer://mycluster <Proxy balancer://mycluster> BalancerMember ajp://server1:8009/cas BalancerMember ajp://server2:8009/cas </Proxy> Hardware High Performance SSL off-load Can be expensive Need multiple devices for HA Software Free (as in Speech & Beer) Very configurable CAS Clustering wiki page recommends session replication You don’t need it Adds complexity Session is only used for storing the webflow state Change WEB-INF/cas-servlet.xml: <flow:executor id="flowExecutor" registry-ref="flowRegistry" repository-type="client"> In service Feb. 2008 – Oct. 2009 Failover Cluster using Heartbeat Default (non-shared) Ticket Registry Apache/Tomcat shared by CAS and Shibboleth IdP Service Registry & Auditing use MySQL Master-Master Replication Location Servers were in the same (poorly outfitted) server room Performance During high-load, CAS & Shibboleth were a bit slow User Experience All tickets were lost on failover, forcing users to login again In production since Oct. 2009 4-node N-to-N Cluster using Heartbeat/Pacemaker Geographically separated (~1KM apart) Memcached Ticket Registry (Repcache) CAS, Shibboleth and other webapps have ‘dedicated’ machines Service Registry & Auditing on dedicated hardware Hardware Load Balancing Off-campus Disaster-Recovery site Currently in Tallahassee Moving it farther North Persistent Ticket Storage ‘Remember Me’ function is highly requested JPA or JBOSS Cache with persistent storage Eric Pierce epierce@usf.edu

but . . it is . . I know you sell . Uribe who . he's far from there . he's done . . . tears and the idea really are . how it'll all the cast of those that don't work for the union is my home . various other provision you again in space stuff . Palm . for Hoover . they finally come . do you think about before even attempt to foster closer that it is here to do some stuff that needs to be done . um . before you can think about that . . still overweight party . we do everything alone . Uribe pretty available which is free software . of the various different outfit every street available for fast . Bozell between ten and the way we have . can you separate now . and historically . Howard said . um . Asia is all about risk . this person you do . when you decided you need to make anything available . his figure out every possible way to fail . and um . he pretty . . he really set down and think about it it's not just bought as does the departure is going to fail . what happened in this week's it is connected to this failed over the network points on power . he really settled with it and figure out what the likelihood of failure is and how long and how extensive is it to six . and Tom . I guess how much will we lose well it's down but in a educational sense of how much of instructional time and . it was . for this can be home loss to think that . and don't forget about all people . the killing has to come . this is actually something done . I'm guilty of the USS time . you and he had the least one back to if not more for everything he did . com . B . the worst thing to happen is all systems are running . period . so he leaves and some some minor thing happens . and nobody can be out of the problem is because they never touched that system . so you really want to get a rotation set where he got more people doing everything building that's looking at different times failures . well what did all the different opposition forces . everything that they knew me to do . and on and giving cases . and . the way you made it all at risk . he is calm . make a list of all me . we fixed that the failures . and some doctors and the Lily Spence a game time . indoor money . Palm . so he dissed down with management and figure out . few things we can fix . this is what we can do . in the next six months to fix this stuff and it doesn't cost any money for a little slow . this is the stuff that we can fix but it's been a top modeling . and this is the stuff that time we don't know how to fix them will have hire somebody to figure out how to move on to work . King . do the stuff in on Wednesday . managements can do . all that when a student friends of Tom . Brocade India Senator gravel team . all are here trying to get into . completely outside the system . they may know . the twenty four . so we need to make sure they know the risks of that . and they're willing to accept it . that's the . overview . the stock even think about when you knew him . each day . but catching particular . Palm . if you run it all and know that the more you use it the more important he comes from . you feel like he sees you for your guests up for you or older . some relatively small effort in trying out new stand in . people start watching on to polio in using signed on time . everybody only was it wants to begin everything . but every service the add on . makes cast that much more important . and that . there is that many more people who are affected . getting to them . and you feel like he says . Serwer She's not enough not from the performance standards and because of calf performed really well . it's from a time perspective . you're going to have partners . so we need more machines that you can take a machine down the Congress' place . our clients or replace whatever time . the new team . for a lot of money out and by machine that you can replace everything online . but there's always going to be something to me she needs for you to be recruited for the last upgrade whatever . so I'm all for machines give you flexibility . . you know it's tough even tougher . were you asked or tax time . this is a really . high level . you know often in these managers are going to an outside source . and your audience are going to be talking often forces . we know . you went . external clients connect . so those are the places that he needs to calm her . as far as I know in those parts that you would notice . reading where it interacts . and with medication . well the sources that cast . text you can a comedian . home directory and tell that are really the same opposition . pretty much time . it's standard practice helpful to many controllers and eighty sites though . that's usually not an issue . com and tell that . Morton After a petition was really easy to sell them on . a specially in common . some commercial tell that you can have . he wrote as always quite so calm and do you remember that the issue . but if you go down obviously you still have water . Castillo all day but if there's nothing except some Africans are still not able to use the service . com . and with love her breasts . it's a way to handle all . on domain controllers and April . Donna let's try one can turn on . Millwood with his back and that's completely up here in advance but there is . don't we stay in the last four or go home for Poland . what are some people do things . that was just one view of the peninsula . armed with services management and . on poverty . storage options are on similar . with Denise and a lap stories for her services from . but as we replicate and obviously you saw him since . the person . four . Palm Desert is management . or this or is this database is Reloaded . I think by phone every two minutes . com . so all in Boston as the No . OK did you sir . some of the new service end . I'll allow access to that time . before that he was only on that particular note of that . on the services and to be new to reload the events . com . and when the party and statistics of the inspectors back in time . he writes that it is for local file on . obviously both of those are optional . but the especially with those on again and Statistics found that something you really . production you really need and the key . don't get on and off . putting information to tell you what . with the ball long . I . he said . Hillary party . wrong . if you've never heard of it . open Source package of Britain who really liked but it runs on the most anything . Palm Beach . he monitors all the genes in the cluster . and get a communication wire . so the scene they can . second the help of Jeannie and . it gives the utility takeover . D virtual IP between be that difficult . the students . Tom . Sims a party that we all noticed any teen use of IDs for Syria with connections when the Kings . sir . if they're located close by . can you see over on hand . no way that no network . com . the possibility of a network failure . the union problems . com . along with the party . there's a cluster resource managers . it included with the party . up to present to Tom . and they're still all rudimentary CRM concluded with the car for a package that they recommend switching peacemaker . um what this year and US U . it kills all of starting and stopping of the service and so if you're running on . yes and actually it'll make sure . yes . home grown on the system is not running . it starts it to sex . the health of thousands who runs a periodic own cells for that . the Jackson Hole . Hmmm . the way we use the party that you guys do . it's all there . and the standard that we all know . some special consideration to capture so it didn't even know that it's been a hell of a cluster bomb . the cluster of her twenty eight . all quiet actually connected . com . we were one term though running on its server . . no dice . on the second scene . recognizes that day . do you see one down . the versa when he moved over to the scene too . all clients are connecting to the virtual machine . Firfer What be so . and they really don't notice of their business . Palm . the one they will lose his that . all of the time to train tickets for all your years service tickets and everything are lost because the search for the senior industry . his only on his team won . it seems here that comes up that's from him . com . so . users up to reopen again . as soon as they look for . . a wall . on one don't see later this was just relief ailing father . because our regional center . we really didn't need one down to because the performance and cast is good enough . day time we had a single machine . everything worked well . we just needed . on the back of having a sale over . . he's really easy to setup menu included in all major one instance for asylum . and they're doing isn't all that history is for sending enough . you can get . Parkin coaster up and going . the fifteen minutes . it's really calm very simple . and you don't have to change the contrast . so . the problems as a Serbia . today begins a search tickets or a loss . and I did vote which interrupted the second year team is completely high and so you're wasting your resources that come back and configuration . so . whatever . today we got a text reports on one bounce or were condoned by the announcers to stick it in front and that fixes it . um . that way . coming next in the local announcers . it's an over to either machine distributor . on the road when the league . so . you're up . you guys . new resource issues anymore everything is good news sometimes . any good . yes itself to get cool announcer . and . this isn't a warmup events . balanced performance there but it would on how not so on . what happened to your home . he could hear the case interesting because when you get you coming . the next two to one . the jurors are thinking . in the morning . three . then when the connection might go to you . you can verify to get that doesn't insist on Tuesday . so I'm out of the media all connections fail . usually not happy . so . and they did cheer together industry . on this . well there's a lease three . freedom then and it's awful . but time . he cheered even registry . basically I'm a single street in both servers or even more and more than two servers . two ten . Siena . I see over popcorn for all of those sitting in coming elections . it doesn't matter because his team one or two . he is allowing all tickets . Tom in the way he does instructions on and said he never did was catch on . and if you're the only instructions did it all works . and she was cast is pretty complex the bomb hit . the cluster cash home . all the changes history to be higher . clusters so you have twenty knows . it will distribute all . but in the fall in the race and if not persistent . everything is in memory . so if all of you know it's going down for whatever reason . you lose all that the couch . you can swap that out for . JVC were flat files for each time . but the default configuration in the configuration that in which he is remembered it . . yes he did he get any notes that diamond ring that will foster them . I do . the unearned . all the instructions are there . no way to boss works . you write a good one . cash server . they immediately broke it's all the other cancers . so your next request . when you read . you can go to any of them . and I think you is one of the matter is for it . I . D . the next person to manage cash which is what we actually is right now . yes . Palm . he was originally designed . I want Journal . the second front in their home ice pollsters . in his cap so . he writes would go to database . but then offered to loan memory in intensity . and down . he reads After that would try to catch first down catch and when will it do to get to the top of my school system . Tom . you don't rule it yourself which note . to store the day on which ones to read from the consulate . it always looks at the same machine . when it tries to grab the kids today . Tom . it's not persistent . everything's a memory . it's really really fast and a protocol is used very simple . it's time . he says the boy didn't have any kind of publication . companies sued a year remaining on protecting network . and then you control all of the No . networks around so they knew that when you see fire along the kids . anybody who can act . they offer the right to own . the wreckage patch time . which is recommended over . to see which he paid for the men can be . the server . after application . Palm . but it hadn't been beating since July . and on . Dean We've seen . we hate for it . on this . Callebs Now we're scientists will air on to the documentation . that is the completion was on this year . back in September . they missed the last story that so . today though . what we heard of that project is in question and there's a new version of men testy . four . I think the catch . Hastert indicated that it still wanted to break . and then he said it still does it still works on . no one wants to still support it . men get from time to drop that so . home . he is . that's him in just a very good and then direct cash from the replication is this continues . the waiters . women cast works . what a right to . our own gift that will give any students are allowed . he writes or one ten eighteen . he really happy that he's going to look at server one for you today . who is the No . the key . sends a bar . there was a three piece today and you know always look at . sir you . the data were one for the first to you today . if you lose . here's the machines . and you lose that day . with cash . you're great . our stubborn and so he's already there but he automatically get through it over . and on the same thing . from searches or one that it's more a master of the cage . com . the team win . he's yes he does and you lose a server . cash is no longer . so the first time they try to connect to Server to you . after details . um and STDs of the Minutemen he climbed faster regenerate . so now the only one server at you . the men to school . they always show up as surreal . so you . Serwer One out all of the victims all replicated over from servers to seventeen . he read to him just being there to serve one will get corrected . com . you don't someone works for two servers . if you extend this out to more than two on one hit only two Server . occasion . it's just me or so . you can't interact for that . conversely more than his or her . but my initial idea was OK I can use for servers and you can go up . breathe . I read it and kind of set up already . for servers . two of them are with you . to the other two . any minute as setup would look at all for servers to one failed to do so on . we would still be available . that doesn't work because of the home . Hatch our them . since then . it rebuilds the hatch after the server to fail . he redistribute all the rights to . time to mold a server to get out . he seems to be . the repetition with more than two servers . Palm . and actually our rhythm doesn't mess with it in a story that it only works with his or her home . D . when that is . if you're going to do more than two servers in your cash in your cast cluster . you have today . the two of them to be in bed . the Manchester . and those who are required to be up . Walsh Oh example of that later on . . . yes . read another genome . . he will love . don't call the data that you know she won . institutions to . com . and so on . but when you think you can change the contrast configuration for which the scenes look at what are certain . Tang . see you . I'm certain you . there was urgency and a cast that the men catch Klein . SEN . Reed tries every year . forget how many seconds it's like once in a midfield redrawn . and then it says it knows is that of its teams up there will be to redistribute catching . Ensor Right . did you get security Street home . spreading today a day so obvious and storage increases persistent . Palmer But the BBC status . you know really wanted . and a loan database and ability . so with the way that everything's on . performance is amazingly good . when I first went to the performance was no substance it's actually countering to death . on DVD pretty well intact . that is meant to be . but with the tradeoffs of getting to write directly to this cannot come to worry about . on both machines to undo them . it's worth the tradeoff . the registry clean com . do you use with the deadlocked where multiple knows tried to run cleaner and elite reservoir same time . and on . watching the entire table not stranded . cleanup will take it . Palm . yes . with touchdowns on three forwards . this Scott said he was earlier . UN presentation of the aisle . this morning . given time . today he says no one is ready to loan him the No . the CDC's of law in strategy . um . basically there's a lot of people in the database and each of the clients after . so here . local one . as you are to stand for . they will . no . . Bryant . . well you along . with those two TD game and he's made the call . . yes I was twenty she is with the the dead dogs before . is that all once you get to a certain calm certain number of tickets . on DVD . the cleanup would take longer and then on . they didn't and you wind up yet . one wins . Bombardier which he is he in . what treaty . reprieve for the nation's teens read recently and they say . he is . yes he was barred . yes Martha here . and the other . and one of the issues that were there were two issues in . um . can you take a step with a single machine . Palm . one was eighty . you lose you . in India . the table . and this generated by Iran . so there was an index of sentiment was . and that's the minimum twenty teams that team . although crack in it . one other one who changed in the area . it's also . what was the transaction . Fortune and Ashland and clean . in time . we are part of the No . yes hello . . no . can we deal with all of those though on . they may need to know what's going on the other servers so the eye . do you see walking the walk people . engineers have to request the law before they're allowed to clean . the registry . on the one certain second inning lead . com . and it nears the expiration time whenever it says a lot . and if that desperation and gone by . on another note can take the law . that wave if you lose and those that had the locks . and one . and his group . four on the clean so they don't . the LA team strategy is just forty two one to lead rounds . yes and a lot people before the winner . Collins So . and beyond . the goal is to you know waffles able to do . cleaner just mark the rooms that . yet the almost three years running the same time begins . Dell on . don't bet on the database . . they soon and we got multiple choices . there's actually not as reasons to do with your car . you catch them . . but these three column . G most cash does he mean by the UN staff that sorry . a bomb expert . that would one go with because you can do . Costello This column . you can say is to of persistent storage . com . a . all around is the best choice if you know I used a mosque . com . that's a pretty good tickets . there's lots of home . Nelson switches on that . and No . in Turkey up so the phone . and also do the scale of rejection of its super he gets to you with the wreckage . pass it . Carolyn up and running in . Crowley an hour . and then apply them . the main cast changes and in cats and urine go on . the status of their cash . from there he is really concerning and he made you look around and alternatives . um . he does . and we've been tested only on to servers running into months . with the GTA . . it's it's found this to be had here . integrity and reliability in reading this . and on . it's . you're using . during past . it's easy for you to get setup com . the DJ database . obviously it's in our part but . most places have a go . the stadium is hurting . in the works for him . introductions into whether he bet on . on some of them this bizarre and frightening . U . it's . latest Al . one seed on . he's in GTA . Kong . there's several people even get lost . com . we use on and actually yes . I think it will . he . the A's improved to second . . over here . the euro . . he . . he . and beyond . and the reason most people didn't actually mean it's very easy to set up . it's yes it's a loser for two machines . yes . it's good . com . the No . it is the is the debris of cash to pay with you . here with that . the incubator . the war . for him . com . it should work but . Tom . I just had it right . no I don't see reason why with more . does he do this Washington out for India . yes . and of the replication . this is it recognizes both get down into the machine and connecting to build on its entire cast of this is the second change . com . well we'll see you Tom . you can buy low ballots are obviously when you can also do it completely with other free software just in the past he can stay and really that's the entire conflict . for doing any of the loan balance . the foster care . one run and that's the end of that . um . it's . really simple home . he doesn't listen in on that . on forty and forty three . and a lot already . so to go in and out . data on the front of life . he was always knew that a month Profit Down Syria . the small configuration . you can have both on both servers up . so in my example former party on . you can have both server running on target . and don't deliver . whenever you too . read my singing again . the New Year period were on the scene one or two . you can be . the City creator of that . they come back off on the server one . and you read Apache running on the front end . and it was never know . Campbell's knows to stay low for too . the server to end on that new patch the market to slow the filter . ten . clients are completely unaware that there was a switch over . so . you do . full mode balancing was completely free software . . the trade off the out years the hardware is any more performance . um . especially since you do the best offload . com . and in . the network and the tricks that the hardware . we're going to do . but resistance and then if you do you mean Cheney down by two of them so that when they lost to No . then the water so twice in the arm . this thing with software is up . it's really your goal on . don't be seeking to use a filter on . traffic for specific IP as they always give to this one certainly got here . you always want make sure you're eating a monumental so . Iraqis from from his office . all we get around it too . seventeen days . com . the fact that I think . . here . Foster stuff with just with free software is due to end in clusters . which is still new party . one or two like he did . you don't want her to a healer . both machines are . she won only controls one for twenty . King You control the other one . you ruined it for what applications on Tuesday . that way they're both fully utilized . but . you don't have all this true . after hitting only . our logo . we're suffering from Santa . if the machines sales . Kathleen just fall over to the other machine and servers to that . two words like he's running on it . you obviously have to do . you obviously have to come . Saddam seems to me you will . no . the nation . there is . configuration to sea with her son was treated in this way with saving time . four five machines . Brady's have their own application of their runs and four applications . and one that's completely spared . and you know if there's a failure on the machine to alter its spare . and spearheading a candle call for calm . in a reduced capacity . but it's clear that once your machine is bigger than any of them . before Bonds . the No . her intent from going forward . we don't . . . was there one question them . you know we age com recommends this area know that contact sessions . what are they sharing between . yes . instances . and you don't really need home . the more he seems to need to make this the call three hundred thousand retired people quiet . um . don't only reason is they are as good though when Lopes state . Palm . and . and ears . Davies got to include me in the meantime . you mentioned there was a possible security issue with this . his solo wanted and that's captured citizens . he's back with new low . see you . the U . one . with that . you would not . so we won't see you . . the national police car . with him . I'm still . yes can top that . so . just to make sure that it's captured on the reporting . the day . controls the phone or use browsers . modern roads are struggling . one . no . and you can be . COL . so anymore . the assassins who are you . you didn't lose time . the Nationals . they're all good defense . well . shootout . yes . . . . . we are casting a bomb . our first cluster free him up in the domain . Collins strictly be over . we see here is you actually are somewhat to the very first of the over Internet into people you can reduce rebound . we are running passing ship . and the reason Nascar after application online poll sign up for pardons for us . and he doesn't look like home . if there was a leader both . and over . Palm . and No . everybody had no warning and since we only had . home to failures and that time there wasn't time to give you everything else was done on schedule maintenance any other treatment and things . I do . he was intending to vote . and Whew . but a . he couldn't say one of the move . Kings one six . also with the performance . Kathleen still on the same machine . D Mohr Carol well the ship . I mean he won't . most creative way to use most of your way tonight expected to be . so we've got . Myron load on . Shula . they are . they pretty much killed ninety seven . no one in two . to fix that . this surgery . St . Palm Center Region are wondering so . twenty minutes actually data centers and can spread in all the new draft would . so all of our current cluster of four Nunez . Palm . yes it's a war knows running on . yesterday you might go . Palm . to be done . who were separated over tyranny in the new closer with the cast . Kassem a machine to one another . on and in a couple of other one doubts that we run on and on . on the other two . Mrs are current . it is honestly quite quite a bit more complex than our whole system . um . he learned here in Daybreak . Aaron . three are in the desert . and if the dolphin color are clues to who . well we begin with . most . and so on . . passes . there was this a machine and we've gotten to the machine here today are clustered there . there are hitting the back court . but their initial key . he has to do . we are . and . when you all for her . undaunted . Palm . or so . from the students from the new float morning to you . D . he's your column . we're looking in . no armor look don't serve . time now for performance are easy but because we are . the movie machines to do that after every site and the one down to work on . we're actually in the process of law . the law into that bill over to your site anything happens for missions . yes . and along with that we're moving the PR side in Tallahassee some work for and worked as a hurricane in a couple Florida . we will be . hello . we had years ago we had four hurricanes that are interesting . yes . two period . so . I . D . they wanted changes under persistent think that because we had a lot of requests from students . the government me function . column . we are . security concerns but that was when serious gains that were done they cannot only allowing that on the key is that going on . they are in one of our wives . so by machine or the functionality they will call home but if you're committed an officer coming in from home . the field will feature No . um . this quarter final . world going home no veto . Arellano her how to run . you heard it here . it's the . the school without progress on the home . that's wrong . Stewart Lou . he sure looks for yourself . the questions . you know . four . we don't know . . the remains of people are you are not and I know . I don't think it is easy . but they did set it up . it doesn't work . you live . they were . s . . and what are you . . . two . all of its own war . . yes sports surrogates . . four . . it is . here are my thing was going to be here and he was leaving . Carlson He didn't we . . . . . she has more . . he . so . and when . it was . . is he . and I will . . . we'll get some misunderstanding . . . . are you . during the two thousand one . the war . . I'm . . you can do that . the lawyers in all he is . if you're going to be roughly separated your connection between his dad . if you're OK . the No . so you don't need to get or they really know about the disease due to a U . he said . yes . plenty on his duty . he's clients . I mean . he has . . . . I want . you are . . I'm great . . . he missed just two pieces missing . but . well the CDC and his team . . that means inconclusive . . he's heard . no . he died . . . Kessler No . no . . you work with them . . . . Q . they're in . really . hello from Daryn to hear you . one . he volunteered and the U . PRI . the loss . . and . Joel you . . . her teacher . we're not . he has delayed the U . Crow . one . Palm . . and the week . . to catch heat . the key to success . two to one loss . . . growth day . the archive . well you are . there were a No . yes . you know . the shooter . . it is . the team . . . you name it . yet the . you will . no . on her . . yes . . Kurds were more creative . and from the going on right now . um . but . Walter Yes . period . column . there are . just be always on the ball in right at us . and Aaron you are . . he knew the law . the DNA . one . I needed to carry on and on . you can come back . . the other . Thompson No . see you . he said . no . a few thoughts with you . and there . s . . . a couple dozen countries . I don't agree more . Paul . . . the the . me .